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Abstract 

This  research  sought  to  determine  whether  a  relationship  exists  between  the  maturity 
level  of  the  software  process,  as  determined  by  the  Software  Engineering  Institute's 
Capability  Maturity  Model  (SEI CMM),  and  the  actual  success  of  particular  software 
projects  at  three  US  Air  Force  Air  Logistics  Centers.  Project  success  is  defined  in  terms 
of  cost,  schedule,  and  quality  criteria.  In  addition,  each  organization’s  acceptance  of  the 
SEI  assessment  as  well  as  its  effectiveness  in  setting  up  follow-on  process  improvement 
programs  were  evaluated.  Finally,  the  process  improvement  programs  were  reviewed 
individually  inasmuch  as  these  programs  are  the  mechanisms  for  achieving  improved 
quality,  lower  cost,  and  on-time  software  projects.  The  research  was  accomplished 
through  a  combination  of  information  gathering  techniques  and  data  analysis.  A  literature 
review,  both  within  and  external  to  the  Department  of  Defense,  summarized  current 
software  process  research,  discussed  the  CMM  in  detail,  presented  case  studies  using  the 
SEI  CMM,  and  introduces  software  project  success  criteria.  Using  information  gathered 
from  three  Air  Force  software  organizations,  we  addressed  each  research  objective. 


AN  EVALUATION  OF  THE 
SOFTWARE  ENGINEERING  INSTITUTE’S 
CAPABILITY  MATURITY  MODEL 
AS  A  FRAMEWORK  FOR 
SOFTWARE  PROCESS  IMPROVEMENT 
AT  THREE  AIR  LOGISTICS  CENTERS 


I.  Introduction 


1.1  General  Issue 

The  budget  for  the  Department  of  Defense  (DoD)  is  getting  smaller  while  the  price 
of  weapon  systems  is  getting  larger.  In  this  environment,  the  DoD’s  reliance  upon 
software  intensive  systems  has  been  increasing  at  an  astounding  rate.  At  the  same  time, 
one  of  the  major  problems  plaguing  the  software  industry  is  late,  over-budget  software. 
This  combination  of  increased  software  reliance,  late  software  delivery,  and  high  software 
price-tags  creates  a  critical  problem  for  DoD  acquisition  and  support  of  software 
dependent  weapons  systems. 

hi  1986,  the  DoD  founded  the  Software  Engineering  Institute  (SEI)  to  develop  a 
methodology  for  characterizing  the  process  used  by  software  organizations  to  develop 
software  products.  The  SEI  methodology  consists  of  a  self-assessment  to  be  used 
internally  by  organizations  and  a  Software  Capability  Evaluation  (SCE)  conducted  by 
trained  evaluators  to  establish  the  current  capability  of  an  organization  to  generate 
soitwarc  md  to  identify  weaknesses  in  existing  processes.  The  evaluation  assesses  the 
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maturity  of  the  functions  in  an  organization  that  contribute  to  the  development  of 
software.  Some  of  these  functions  include  project  management,  configuration 
management,  training,  software  quality  assurance,  and  automation.  At  the  conclusion  of 
the  Software  Capability  Evaluation,  a  level  of  process  maturity  (I {lowest}  through 
V{ highest}),  reflecting  the  overall  maturity  of  the  organization  as  a  software  producer,  is 
assigned  to  the  organization  . 

Whether  software  is  developed  internally  by  Air  Force  organizations  or  produced  by 
contractors,  the  software  development  process  must  be  controlled  and  managed  properly 
to  gain  improvements  necessary  to  maximize  future  Air  Force  investments.  By  studying 
Air  Force  software  development  organizations  that  have  used  the  SEI  methodology,  we 
may  be  able  to  determine  if  a  relationship  exists  between  successful  software  development 
and  support  projects  and  SEI  assessment  levels.  If  the  SEI  rating  proves  to  be  a  sound 
predictor  of  project  success,  it  may  be  a  valuable  aid  in  reducing  persistent  problems  with 
late,  over-budget  software  within  the  Department  of  Defense  and  private  industry.  This 
relationship  should  indicate  whether  the  SEI  rating  is  an  accurate  predictor  of  software 
project  success. 

1.2  Specific  Problem 

A  number  of  Air  Force  software  organizations  have  undergone  the  SEI  self- 
assessment  during  the  past  few  years.  However,  no  information  is  available  on  the 
relationship  between  the  resultant  maturity  level  and  the  ability  of  an  organization  to 
produce  quality,  maintainable  software.  Studies  of  a  few  defense-related  software 
contractors  seem  to  suggest  that  a  higher  maturity  level  is  related  to  increased  efficiency 
and  software  generation  capability  (7,  8, 14, 16).  An  accurate  evaluation  of  the 
relationship  between  the  Capability  Maturity  Model  (CMM)  levels  and  the  success 
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organizations  have  in  producing  quality,  on-time,  within-budget  software  is  needed, 
however,  to  establish  the  CMM's  value  to  the  Air  Force.. 

1.3  Research  Objectives 

By  studying  organizations  that  have  used  the  SEI's  Capability  Maturity  Model  to 
assess  their  software  processes,  we  determined  if  a  relationship  existed  between  maturity 
level  and  project  success.  In  addition,  we  evaluated  the  impact  of  the  assessments  and 
follow-on  process  improvement  programs  on  the  assessed  organizations  to  determine  the 
overall  effect  on  the  organization. 

1 .4  Scope! Limitations 

We  chose  a  case  study  methodology  to  limit  the  scope  of  this  research  to  a 
manageable  effort  within  our  time  constraints.  Selection  of  three  similar  cases  enabled  us 
to  study  each  case  in  depth  and  provided  results  that  can  be  further  generalized  Further 
discussion  of  the  case  study  methodology  is  contained  in  chapter  three. 

Candidate  case  study  organizations  were  limited  to  Air  Force  organizations  that 
have  been  evaluated  at  least  once  by  the  SEI CMM.  Results  from  these  cases  can  be  used 
not  only  by  the  case  study  organizations,  but  can  also  serve  as  background  information  or 
a  guide  for  other  Air  Force  software  organizations. 

Although  there  are  many  different  definitions  of  project  success,  we  chose  to  define 
project  success  by  the  combination  of  cost,  schedule,  and  quality  (customer  satisfaction). 
Justification  for  this  definition  is  discussed  in  section  2.3. 

1.5  Overview 

This  chapter  has  outlined  the  problem  and  importance  of  the  research  to  be 
undertaken.  In  addition,  we  have  limited  the  scope  of  the  thesis  to  studying,  via  case 
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study,  organizations  that  have  been  assessed  using  the  SEI CMM.  The  research  objectives 
have  also  been  briefly  stated  as:  1)  Determine  whether  a  relationship  between  maturity 
level  and  project  success  exists.  2)  Examine  the  managerial  and  worker  level  reaction  and 
perception,  within  assessed  organizations,  to  the  SEI  CMM  process  assessment  and 
follow-on  process  improvement  efforts  for  the  purpose  of  establishing  whether  the 
environment  was  conducive  to  an  effective  process  improvement  program.  3)  Review 
organizational  process  improvements  following  the  assessment(s)  because  a  process 
improvement  program  is  the  mechanism  for  achieving  improved  quality,  lower  cost,  and 
on-time  software  projects. 
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n.  Literature  Review 


2.1  Introduction 

The  importance  of  software  quality  to  the  Department  of  Defense  cannot  be 
understated.  This  literature  review  discusses  tl  ^  current  state  of  software  development, 
acquisition,  modification,  and  maintenance.  Following  this  introduction,  the  second 
section  briefly  explains  die  processes  used  to  build  and  support  software.  The  quality  of  a 
software  product  is  also  discussed  here  as  an  element  of  the  process  used  to  build  and 
maintain  it.  Project  success  in  the  software  industry  is  discussed  in  the  third  section  of  this 
chapter.  Sections  four  and  Five,  respectively,  introduce  the  Software  Engineering  Institute 
and  its  Capability  Maturity  Model.  After  these  explanations,  some  cases  demonstrating 
the  value  of  the  Software  Engineering  Institute's  approach  are  summarized  in  section  six. 
Section  seven  outlines  present  DoD  use  of  the  SEI  methodology  and  presents  some 
expectations  concerning  future  use  of  the  SEI  methodology  within  the  DoD  and  civilian 
software  markets. 

2.2  Software  Process 

A  software  process  is  simply  a  method  used  to  develop,  modify,  or  maintain 
software.  One  example  is  the  waterfall  model  (Figure  1),  which  begins  with  determining 
the  requirements,  proceeds  to  analysis  and  design  of  the  system,  and  continues  with 
coding  the  system,  testing  the  system,  and  finally  implementing  the  software.  With  the 
simple  waterfall  model,  the  events  are  entirely  sequential.  More  complex  models  have 
been  developed  in  order  to  perform  tasks  that  can  be  done  at  the  same  time.  In  some 
cases,  for  example,  coding  and  test  development  can  occur  simultaneously. 
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Figure  1.  Waterfall  Model  of  Systems  Development  (21:83) 

2.3  Project  Success  and  Quality 

In  order  to  determine  whether  the  SEI’s  Capability  Maturity  Model  (CMM)  is  a 
useful  tool,  we  must  understand  what  constitutes  project  success  in  the  software  realm.  In 
general,  "successful  projects  are  judged  to  be  those  doing  better  than  average  on  the 
cviteria  of  cost,  schedules,  and  the  satisfaction  of  key  project  participants  (client,  project 
manage*-,  project  team,  and  syste:  .  uevelopment  organizations).  Other  criteria  include 
follow-up  work,  end-user  satisfaction,  end-item  quality,  and  meeting 
specifications"  (17:472).  In  short,  project  success  is  defined  as  meeting  a  pre-defined 
schedule  within  cost  while  simultaneously  achieving  the  technical  requirements  of  the 
project. 

The  premise  of  software  process  improvement  is  that  product  quality  or  project 
success  is  directly  related  to  the  quality  or  maturity  of  the  software  process.  "All  failures 
are  system  failures  in  the  sense  that  they  are  actually  the  output  of  a  particular  system. 
That  is  to  say  that  there  are  features  or  defects  in  the  system  which  produced  or  allowed 
^that  failure"  (17:463).  This  implication  that  the  process  is  responsible  for  the  failure  of  a 
project  is  especially  true  of  complicated  software  development  efforts  where  a  large 
amount  of  the  effort  is  expended  in  configuration  management,  design,  and  the 
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implementation  of  a  supportable  system  that  conforms  to  rigid  programming  and 
documentation  standards. 

"The  software  process  is  the  set  of  tools,  methods,  and  practices  we  use  to  produce 
a  software  product.  The  objectives  of  software  process  management  are  to  produce 
products  according  to  plan  while  simultaneously  improving  the  organization's  capability  to 
produce  better  products"  (12:3).  The  basis  for  the  SEI's  CMM  is  to  assess  the  software 
process,  recommend  changes,  and  guide  improvement  efforts  resulting  in  more  successful 
projects.  Each  level  in  the  CMM  has  specific  requirements  that  should  result  in  a  better 
software  process,  and  consequently  more  successful  projects. 

Process  improvement  in  the  civilian  software  arena  is  undertaken  at  great  expense 
with  the  express  goal  of  producing  higher  quality  software  in  a  more  efficient  manner. 
Other  benefits  include  developmental  cost  savings  and  long-term  support  cost  reduction 
due  to  increased  initial  product  quality.  According  to  Hersh,  product  quality  should  be  the 
focus  of  all  process  improvement  efforts  (9:12).  In  addition,  Hersh  points  out  that  many 
professionals  confuse  the  distinction  between  process  change  and  improvement  (9:12). 
Finally,  he  advises  organizations  embarking  upon  process  improvement  to: 

(1)  select  a  model  for  defining  the  process  and  relevant  data, 

(2)  collect  relevant  data  in  a  uniform  and  consistent  manner,  and 

(3)  evaluate  and  refine  based  upon  detailed  analysis  of  this  data.  (9:12) 

Curtis  suggests  two  separate  measures  of  maturity  that  differentiate  between  product 
quality  and  process  quality.  First,  internal  maturity,  the  type  of  measure  used  by  the 
Software  Engineering  Institute,  identifies  an  "organization's  engineering  and  management 
practices"  (5:89).  The  second  measure,  external  maturity,  is  the  user  s  view  of  the 
maturity  of  the  product  and  the  organization  based  on  experience  with  the  product  (5:89). 
Regardless  of  the  process  used  to  develop  the  software,  the  quality  of  the  product 
determines  the  user's  view  of  the  organization.  This  is  certainly  true  for  the  customer  who 
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buys  off-the-shelf  software  for  home  computers.  It  is  also  true  for  the  purchase  of  large 
defense  related  software  projects,  although  costs  and  schedule  play  a  larger  part  in  the 
latter's  consideration  because  the  customer,  the  DoD  in  this  case,  is  more  directly  aware  of 
the  production  or  support  effort. 

2.4  Software  Engineering  Institute 

The  Department  of  Defense  sounded  a  note  of  urgency  for  improvement  of  the 
software  development  process  when  it  founded  the  Software  Engineering  Institute  (SEI) 
in  1986  (10:30).  The  SEI  is  located  at  Camegie-Mellon  University  in  Pittsburgh, 
Pennsylvania.  One  of  the  goals  of  the  SEI  is  to  "develop  and  refine  a  process  framework 
and  assessment  methodology  for  characterizing  the  processes  used  by  software 
organizations  to  develop  or  evolve  software  products"  (1 1:277).  The  framework 
produced,  called  the  Software  Process  Maturity  Model,  is  based  on  the  premise  that  "the 
quality  of  a  software  product  is  largely  governed  by  the  quality  of  the  process  used  to 
create  and  maintain  it"  (11:278).  In  addition,  the  paradigm  assumes  that  "the  process  of 
producing  and  evolving  software  can  be  defined,  managed,  measured,  and  progressively 
improved"  (11:278). 

The  SEI  assessment  methodology  consists  of  a  self-assessment  for  internal  use  and  a 
"Software  Capability  Evaluation  (SCE)  used  by  government  agencies  to  judge  how 
capable  companies  are  at  developing  software"  (3:25).  Because  of  its  close  relation  to  the 
SCE,  the  internal  software  process  assessment  has  become  valuable  beyond  its  intended 
ability  to  help  manage  the  quality  of  a  software  development  process.  Companies 
frequently  use  this  voluntary  self-assessment  as  a  preparatory  test  prior  to  a  government 
required  SCE.  Many  Department  of  Defense  agencies  currently  require  capability 
evaluations  prior  to  major  software  contract  awards  in  order  to  weed  out  low-scoring 
organizations  from  the  bidding  process  for  government  contracts. 
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2.5  SE1  Capability  Maturity  Model 


According  to  the  director  of  the  SEI  software  process  program,  "the  SEI  maturity 
model  is  based  on  quality  principles  that  have  been  widely  proven  in  both  engineering  and 
manufacturing  and  are  now  being  demonstrated  as  equally  effective  for  software"  (13:45). 
These  quality  principles  predicate  that  the  stages  of  the  software  development  process 
must  be  clearly  identified.  Treating  software  development  like  an  assembly  line  or  factory, 
the  SEI  process  model  assumes  that  once  the  process  is  defined,  it  can  be  replicated  and 
will  respond  to  the  same  quality  control  management  and  improvement  ideas  that  apply  to 
producing  sheets  of  copper  in  an  industrial  factory  (3:35). 

To  put  this  theory  into  practice,  the  software  process  maturity  model  identifies  five 
levels  of  process  maturity  (table  1).  Each  level  is  based  on  the  capabilities  of  lower  levels 
and 

•  represents  an  historical  phase  of  evolution  for  a  software  organization, 

•  represents  a  reasonable  measure  of  improvement  to  achieve  from  the 
prior  level, 

•  suggests  interim  improvements  goals,  and  progress  measures,  and 

•  makes  obvious  a  set  of  immediate  improvement  priorities  once  an 
organization's  status  in  the  framework  is  known.  (14:14) 

At  Level  I  of  the  model,  the  organizational  process  operates  without  formal 
procedures  (1 1:279).  The  simplest  example  of  this  is  a  small  "mom  and  pop"  operation 
that  consists  of  talented  programmers  designing  software  as  they  write  it.  This  ad  hoc  or 
chaotic  process  can  also  exist  in  larger  organizations  with  defined  procedures  but  without 
management  controls  to  enforce  those  procedures.  The  most  of  software  producing 
organizations  operate  at  this  initial  level,  although  a  few  advanced  organizations  have 
scored  much  higher.  The  model  does  not  contend  that  success  is  not  possible  in  this  type 
of  organization,  it  just  claims  that  success  is  less  likely  and  more  dependent  on  key 
personnel. 


Table  1 


SE1  Software  Process  Maturity  Model  (11:279, 14:14) 


Level 

Characteristics 

Key  Challenges 

Resulting 

Risk 

Productivity 
And  Quality 

5.  Optimizing 

Improvement 
fed  back  into 
process 

V  Ataomaooc 

Extremely 

Low 

Very  High 

4.  Managed 

(Quantitative) 

Measured 

Process 

V  Changing  technology 

V  Problem  analysis 

V  Problem  prevention 

Low 

High 

3.  Defined 

(Qualitative) 
Process  defined 
and 

institutionalized 

V  Process  measurement 

V  Process  analysis 

V  Quantitative  quality  plan 

Moderate 

Moderate 

2.  Repeatable 

(Intuitive) 
Process 
dependent  on 
individuals 

V  Tram  mg 

V  Technical  practices  - 
reviews  end  testing 

V  Process  focus  -  standards. 

process  troops 

High 

Low 

1.  Initial 

(ad  hoc/chaotic) 

V  Project  management 

V  Project  pianang 

V  Coafif  unban  management 

V  Software  outlay  assurance 

Extremely 

High 

Very  Low 

Level  D  organizations  have  established  basic  project  controls  and  are  capable  of 
repeating  prior  successes  with  similar  projects.  New  challenges  represent  great  risks  to 
these  organizations  because  they  have  continual  quality  problems  and  lack  a  systematic 
framework  for  improvement  (11 :279). 

In  order  to  move  from  Level  II  to  Level  ID,  an  organization  must  define  its  standard 
software  process  architecture.  In  addition,  a  software  engineering  process  group  must 
exist  to  lead  process  improvement.  Despite  having  a  defined  process,  key  challenges, 
including  process  measurement  and  analysis,  remain. 
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Progression  to  Level  IV  depends  on  an  organization's  ability  to  examine  and  improve 
its  development  process.  Unlike  the  examination  and  process  improvement  possible  at  the 
defined  level,  the  managed  level  implies  the  ability  to  measure  the  effectiveness  of 
improvements  to  the  process. 

Automated  data  gathering  (as  opposed  to  manual  data  gathering  which  is  subject  to 
bias)  on  process  improvements  is  one  of  two  key  differences  that  distinguish  the 
optimizing  level  from  the  managed  level.  The  second  distinction  between  the  two  highest 
levels  is  that  at  the  optimizing  level,  "management  redirects  its  focus  from  the  [software] 
product  to  [the  software  development]  process  analysis  and  improvement"  (1 1:279). 

In  order  to  receive  any  value  from  this  framework,  a  company  must  know  its 
position  in  the  structure.  The  self-assessment  and  capability  evaluation  determine  status 
within  the  framework  through  questionnaires,  interviews,  and  documentation  reviews. 

The  questionnaires  are  divided  into  key  process  areas  that  represent  the  important 
requirements  of  each  level  of  the  model.  Each  level  in  the  CMM  is  assessed  by  a 
questionnaire  consisting  of  four  to  33  questions  and  covering  each  of  the  relevant  key 
process  areas  in  each  level. 

Because  there  is  no  minimum  requirement  to  be  declared  a  Level  1  organization,  this 
plateau  contains  a  wide  range  of  organizational  abilities.  Critics  point  to  the  fact  that 
although  an  organization  exhibits  some  of  the  characteristics  of  a  Level  IV  process,  its 
failure  meet  the  requirements  of  Levels  II  and  III  results  in  a  Level  I  rating  (3:31).  The 
SEI  justifies  this  rating  process  by  arguing  that  some  of  the  higher-level  benefits  are 
meaningless  within  an  immature  process  with  no  proper  foundation  to  build  upon  (13:46). 
An  analogy  to  this  lack  of  foundation  is  having  a  super  computer  on  your  desk  without  the 
knowledge  to  use  it  for  anything  but  word  processing. 

In  February  1993  the  SEI  released  version  1.1  of  the  CMM.  Although  no  significant 
changes  to  the  basic  structure  of  the  CMM  had  been  made,  there  were  several 
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enhancements  and  wording  changes.  "Most  of  the  changes  we  made  to  CMM  version  1.0 
wens  done  to  improve  the  consistency  of  the  key-practices  structure,  clarify  concepts,  and 
provide  consistent  wording"  (18:19). 

The  most  significant  change  is  the  goal  oriented  approach  used  to  satisfy  each  key 
process  area.  "We  rewrote  all  the  goals,  to  emphasize  the  process  end  states  rather  than 
results,  and  to  remove  subjective  words  like  'effective.'  Each  key  practice  maps  to  one  or 
more  goals,  and  each  goal  and  its  associated  practices  can  be  considered  a  subprocess 
area.  Satisfying  all  the  goals  satisfies  the  key  process  area"  (18: 19).  This  change  serves  to 
guide  an  organization  clearly  through  process  improvement  by  providing  specific  goals 
that,  when  met,  satisfy  key  process  areas  which  in  turn  lead  to  a  higher  maturity  level. 

2.6  Assessment  and  Evaluation  Case  Studies 

Several  cases  have  been  reviewed  here  to  illustrate  how  the  SEI  CMM  has  been  used 
by  organizations  to  evaluate  their  software  development  processes.  The  purpose  of  the 
discussion  is  to  show  that  the  CMM  is  a  useful  assessment  and  process  improvement 
guide.  In  addition,  since  each  of  these  organizations  is  civilian  (although  defense-related), 
differences  between  their  process  assessment/improvement  experiences  and  those  of  Air 
Force  organizations  may  become  evident.  Our  literature  search  revealed  only  examples  of 
successful  implementations  of  the  SEI's  CMM.  That  is  not  to  say  that  the  SEI’s  CMM 
always  results  in  success,  only  that  no  published  examples  to  the  contrary  were  found. 

2.6.1  Hughes  Aircraft.  The  Software  Engineering  Division  (SED)  of  Hughes 
Aircraft  in  Fullerton,  California  underwent  SEI  process  assessments  in  1987  and  1990  and 
provides  an  excellent  example  of  a  successful  application  of  the  SEI  software  process 
model.  Hughes  SED  requested  and  paid  $45,000  for  the  initial  assessment  by  the  SEI. 
During  each  assessment,  the  team  reviewed  questionnaires,  interviewed  key  players,  and 


12 


verified  documentation  for  approximately  six  projects.  The  1987  assessment  graded 
Hughes  a  Level  D  organization  (14:1 1).  The  assessment  team  recommended 

•  establishing  a  centralized  database  on  cost  estimates,  cost  experience, 
and  schedule  performance, 

•  establishing  uniform  data  definitions  across  projects,  and 

•  providing  the  resources  needed  and  the  responsibility  assignments 
required  for  gathering,  validating,  entering,  accessing,  and  supporting 
the  projects  in  analyzing  this  data.  (14:13) 

These  recommendations  are  consistent  ./ith  the  shortcomings  of  most  Level  II 

organizations.  Hughes  agreed  with  these  recommendations  and  implemented  an  action 

plan  to  correct  their  deficiencies.  This  action  plan  cost  approximately  $400,000  and  78 

man-months  of  labor  over  the  next  two  years  (14).  As  a  result  of  this  effort,  the  1990 

assessment  found  Hughes  SED  tc  be  a  strong  Level  IE  organization.  Once  again  the 

recommendations  were  consistent  with  the  key  challenges  of  the  defined  level. 

Hughes  SED  earned  the  benefits  of  improved  working  conditions,  employee  morale, 
and  performance  in  terms  of  project  schedules  and  costs.  In  fact,  "Hughes  estimates  the 
resulting  annual  savings  to  be  about  $2  million"  (14: 1 1).  Obviously,  from  Hughes  SED’s 
point  of  view,  the  assessments  and  their  associated  costs  are  well  worth  it. 

2.6.2  Westinghouse  Electronic  Systems.  Westinghouse  Electronic  Systems 
Group  (ESG)  also  participated  in  an  SEI  assessment  Because  the  results  of  an 
assessment  are  confidential,  Westinghouse  ESG  chose  not  to  publicize  their  numerical 
rating,  but  did  publish  the  "primary  recommendation  from  the  assessment,”  which  was  to 
"review  ESG's  existing  software  engineering  process  group  activities  against  the  SEI’s 
charter  and  functions  and  make  appropriate  recommendations  for  improvement" 
(16:1582). 

We  can  assume  Westinghouse  was  rated  Level  II  because  the  improvement 
recommendations  represent  the  key  challenges  for  that  level.  In  this  case,  Westinghouse 
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already  had  a  software  engineering  process  group,  but  it  had  failed  to  completely  perform 
the  functions  normally  associated  with  an  SEPG.  Reasons  for  their  failure  include 
organizational  constraints  and  a  lack  of  understanding  of  the  SEPG’s  function.  The 
Westinghouse  report  points  out  that  virtually  all  SEPG  functions  were  being  performed 
somewhere  in  the  electronic  systems  group,  but  many  were  not  specifically  assigned  to  the 
SEPG.  To  alleviate  this  problem,  Westinghouse  created  charters  for  each  component  of 
their  SEPG.  In  addition,  Westinghouse  ESG  satisfied  all  SEPG  requirements  by  creating 
new  technology  groups  for  education,  requirements,  and  metrics  (16:1584). 

Overall  the  SEI  model  was  implemented  successfully.  While  still  in  the  process  of 
implementing  their  action  plan,  Westinghouse  demonstrated  that  its  ability  to  work  in  new 
project  areas  such  as  Ada  and  CASE  (Computer  Aided  Software  Engineering)  tool  visibly 
improved  (16:1584) 

2.63  Raytheon  Equipment  Division.  Raytheon  is  another  defense  related 
corporation  that  implemented  an  SEI  process  improvement  program.  In  1988,  Raytheon 
began  a  software  engineering  initiative  through  an  SEI-style  process  assessment.  The  in- 
house  reviewers  rated  the  corporation  at  the  initial  (ad  hoc)  level  and  identified  four  key 
areas  for  process  improvement.  Many  software  professionals  were  involved  in  the  process 
improvement  program  on  a  part-time  basis.  This  strategy  helped  to  increase  ownership  of 
changes,  reduced  the  need  to  re-solve  problems  already  corrected  by  pans  of  the 
organization,  and  smoothed  the  implementation  of  changes  within  on-going  projects 
(7:83). 

Raytheon  is  proud  to  say  the  improvements  are  already  paying  off.  Using  Phil 
Crosby's  Cost  of  Quality  idea,  Raytheon  identified  four  costs  categories  for  software: 
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•  Performance  costs  are  those  associated  with  doing  it  right  the  first  time. 

•  Non-conformance  costs  are  rework  costs. 

•  Appraisal  costs  are  associated  with  testing  the  product. 

•  Prevention  costs  are  incurred  in  attempts  to  prevent  faults.  (7 :84) 

The  sum  of  the  final  three  cost  categories  is  called  the  "cost  of  quality"  (7:84). 

Raytheon's  initiative  increased  prevention  costs,  had  virtually  no  effect  on  appraisal  costs, 
and  dramatically  reduced  non-conformance  costs.  By  1992,  Dion  estimated  $9.2  million 
in  savings  as  a  direct  result  of  the  initiative  (7:84).  In  1993,  the  savings  were  even  more 
impressive.  Additional  projects  added  to  the  original  study  increased  the  estimated 
savings  to  $15.8  million  (8:28). 

In  addition  to  cost  savings,  the  initiative  has  an  impressive  list  of  accomplishments: 

•  a  $7.70  return  on  every  dollar  invested  (8:28). 

•  a  two-fold  increase  in  productivity  (8:28). 

•  an  evolution  from  Level  I  on  the  SEI  maturity  scale  (through  Level  II)  to 
the  defined  level  (Level  HI)  (8:28). 

•  Increased  business  justifying  a  25  percent  personnel  increase  (8:35). 

During  the  five-year  initiative,  nearly  $1  million  has  been  invested  in  process 

improvement.  Where  late,  over-budget  software  was  the  norm  in  1988,  Raytheon 
increasingly  achieves  early  and  under-budget  software  efforts  (8:35).  In  fact,  early 
delivery  bonuses  are  not  even  counted  in  the  costs  savings  for  the  process  improvement 
initiative. 

Obviously,  Raytheon  is  pleased  and  proud  of  their  success.  Top  management  has 
been  convinced  that  the  process  improvement  climb  up  tire  SEI  maturity  ladder  more  than 
pays  for  itself.  Dion  credits  much  of  the  success  to  management  commitment  to  and 
employee  ownership  of  the  software  engineering  initiative. 


2.6.4  NASA.  NASA  is  also  exploring  the  benefits  the  software  process  maturity 
model  has  for  their  mission-critical  software.  NASA  had  two  teams  trained  by  the  SEI  to 
perform  assessments.  Once  trained,  they  decided  to  "perform  an  SEI  contractor  capability 
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evaluation  on  the  Space  Shuttle  flight  software  program"  (8:299).  Surprisingly,  the 
contractor  was  rated  at  the  optimizing  level  (Level  V).  Of  course,  this  did  not  mean  there 
was  no  room  for  improvement.  In  fact,  the  team  identified  entry-level  training, 
consolidation  of  formal  inspection  data,  and  quality  assurance  procedures  and  testing  as 
areas  for  potential  improvement  (4:299). 

To  get  the  complete  picture  of  contractor  performance,  NASA  also  wanted  the 
ability  to  test  contract  management  and  software  quality  assurance  organizations.  As  a 
result  of  these  requirements,  they  decided  to  adapt  the  SEI  contractor  capability 
evaluation  "for  evaluating  contractors  developing  NASA  software  and  for  NASA- 
developed  software"  (4:300).  At  this  time  the  software  quality  assurance  has  been 
developed,  tested,  and  fielded.  Four  major  problems  were  found  in  each  NASA  center: 

•  No  center-wide  software  assurance  policies  or  standards 

•  Very  little  or  no  software  quality  assurance  training 

•  No  procedures  to  ensure  that  specific  software  assurance  requirements 
were  specified  on  internally  developed  software  or  on  contracts 

•  No  recourse  for  independent  software  assurance  personnel  to 
document  and  resolve  problems  early  in  development.  (4:302) 

NASA  still  plans  to  develop  a  model  for  evaluating  contract  or  acquisition 
management  capabilities.  Early  success  with  the  software  assurance  model  suggests  the 
SEI  model  can  be  adapted  for  contract  management  also.  In  fact,  recently-completed 
research  by  Captain  Summers  and  Captain  Dickerhoff  of  the  Air  Force  Institute  of 
Technology  outlines  a  prototype  model  that  evaluates  the  ability  of  an  Air  Force 
organization  to  acquire  software  systems  through  the  contracting  process  (6). 

2.7  Viability  of  the  SEI  Model 

2.7.1  Private  Sector.  While  the  SEI  program  was  developed  primarily  for 
software  developers  associated  with  the  Defense  Department,  no  constraints  limit  the 
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model  to  this  group.  In  fact,  this  chapter  has  already  discussed  other  government  agencies 
such  as  NASA  (4)  using  th,  SEI  system.  According  to  Bollinger  and  McGowan,  even 
private  industry  is  likely  to  use  the  SCE  ratings  "based  on  the  assumption  that  a  rating 
system  that  is  good  enough  for  the  US  government  should  be  good  enough  for  them  too" 
(3:26). 

Earlier  research  by  David  Jobber  and  others  showed  that  75  percent  of  software 
consumers  said  they  would  choose  software  with  a  quality  certificate  in  preference  to 
software  without  certification  (15:24).  Carrying  this  into  the  present,  we  can  be  fairly 
certain  that  the  same  would  hold  trie  for  choosing  software  with  or  without  a  SEI  process 
capability  evaluation. 

Another  reason  to  believe  the  SEI  model  will  spread  is  that  current  figures  show  that 
85  percent  of  all  organizations  that  have  undergone  SEI  capability  evaluations  have  been 
rated  Level  I.  Only  13  percent  were  rated  Level  II  and  a  smaller  0.9  percent  were  rated 
Level  III  (1 1:282).  This  low  average  maturity  agrees  with  Humphrey’s  assertion  that  "not 
enough  attention  is  paid  to  the  overall  software  development  process  itself'  (i0:28).  He 
also  states  that  the  ad  hoc  approach  currently  in  use  by  most  software  development 
companies  "  will  not  be  sufficient  to  utckle  the  task  of  developing  complex  software 
systems  for  today  and  tomorrow"  (10:28). 

The  relative  immaturity  of  the  software  engineering  field  is  also  fuel  for  the  argument 
that  use  of  the  SEI  model  will  spread.  Until  now,  all  organizations  were  competitive 
because  nearly  all  organizations  were  operating  at  the  initial  level.  Soon,  nowever, 
dedicated  companies  using  the  SEI  approach  will  begin  to  show  the  dramatic  improvement 
demonstrated  by  Hughes  and  Raytheon  (14,  7,  8).  No  longer  able  to  compete,  inefLcient 

£ 

Level  I  companies  will  either  go  out  of  business  or  begin  to  use  the  newly  established 
software  engineering  tools  including  the  SEI  software  process  model. 
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2.7.2  Department  of  Defense.  Although  the  private  sector  use  of  the  SEI 
capability  maturity  model  appears  to  be  on  the  upswing,  the  Department  of  Defense  and 
its  associated  contractors  are  currently  and  will  probably  continue  to  be  the  largest  group 
to  utilize  the  Capability  Maturity  Model  (CMM).  Within  the  Air  Force  and  the  Navy, 
ratings  are  already  being  used  as  part  of  contract  source  selections  (2,  19:36). 

The  Navy  estimates  that  the  SEI's  software  capability  evaluations  have  been  used  in 
"more  than  20  acquisitions  since  late  1987,  some  of  them  involving  contracts  worth  more 
than  $100  million"  (19:36).  Much  like  the  waterfall  model,  defense  acquisitions  follow  a 
time-phased  process.  The  process  used  for  selecting  the  contractor  for  the  three-year,  $95 
million  software  avionics  contract  is  presented  in  figure  2. 

The  key  to  making  these  steps  work  with  the  DoD  acquisition  process  is  to  involve 
the  evaluation  team  early-in  the  planning  stages.  The  Naval  avionics  acquisition  program 
regularly  uses  the  SCE  on  contracts  that  include  major  software  efforts  (19).  While  the 
Air  Force  acquisition  process  is  similar  to  the  Navy’s,  the  Air  Force  has  yet  to  incorporate 
the  SCE  on  a  regular  basis.  A  similar  methodology  within  the  Air  Force  is  Aeronautical 
Systems  Center’s  (ASC)  Software  Development  Capability/Capacity  Review  (SDCCR) 

(1).  This  review  is  used  by  n.^,2  for  major  software  acquisition  efforts  and  parallels  the 
Software  Capability  Evaluation  (SCE)  used  in  the  Naval  acquisition  case. 

The  evaluation  team  members  must  also  be  trained  in  the  evaluation  process.  Once 
the  proposals  have  been  received,  the  team  must  assist  in  the  evaluation  of  proposals  and 
selection  of  bidders  worthy  of  on-site  evaluations.  The  on-site  review  requires  careful 
planning  in  selection  of  projects  to  be  reviewed,  team-member  roles,  and  reviewing 
proposal  information  for  further  questions.  In  the  Naval  avionics  acquisition  case,  one  of 
the  lessons  learned  was  the  need  for  more  preparation  time  prior  to  the  on-site  evaluation. 

Since  the  Navy  proj  't  was  software  intensive,  the  evaluation  was  weighted  heavily 
in  the  source  selection.  In  fact,  the  evaluation,  which  counted  for  one-third  of  the  overall 
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score  for  each  bidder,  was  by  far  the  largest  factor  and  significantly  penalized  those 
bidders  who  failed  to  complete  the  request  for  proposal  questionnaires  (19).  According  to 
the  evaluation  team  leader,  Rugg,  the  evaluation  process  was  beneficial  to  the  Navy  in  the 
selection  process  and  also  each  contractor  involved.  Managers  at  both  winning  and  losing 
contractors  commented  on  the  evaluations’  value  (19:45). 


Figure  2.  SCE  Process  and  Time  Line  for  Avionics  Contract  Award  (19:37) 
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2.8  Summary 


The  Software  Engineering  Institute's  software  process  maturity  model  is  based  on 
the  notion  that  the  software  product  can  only  be  as  good  as  the  process  used  to  develop  it. 
The  SEI  model  uses  accepted  quality  improvement  principles  applied  to  software 
problems.  For  practical  purposes,  the  five  tiered  rating  system  enables  managers  to 
determine  where  they  stand  within  the  framework.  The  ratings  can  be  determined  through 
in-house  assessments  and  through  capability  evaluations.  Capability  Evaluations  are 
currently  used  by  many  government  agencies  to  aid  in  the  contracting  process. 

Early  assessments  and  evaluations  have  demonstrated  utility  when  used  in 
combination  with  diligent  action  plans  based  on  identified  problem  areas.  Despite  the 
effort  and  expense  required  to  undertake  these  assessments  and  associated  improvements, 
Hughes  Aircraft  and  Raytheon  Equipment  Division  have  shown  the  process  to  be 
financially  beneficial  (14,  7,  8).  Besides  evaluating  software  development  capability,  this 
framework  can  be  adapted  successfully  for  use  in  software  assurance  and  software 
contract  management  capability. 

The  potential  impact  of  this  model  extends  far  beyond  the  bounds  of  the  Department 
of  Defense.  This  far-reaching  potential  emphasizes  the  need  for  further  research  into  the 
model's  success  in  specific  software  efforts.  Low  world- wide  software  capability  levels 
demonstrate  the  need  for  process  improvement  strategies.  In  addition,  the  sales  value  of 
this  type  of  certification  is  proven  in  the  public  marketplace  and  is  evolving  with  the 
Department  of  Defense.  Companies  that  intend  to  be  technically  sound  a  successful  in 
the  late  1990s  will  undoubtedly  require  good  SEI  capability  ratings. 


HI.  Methodology 


3.1  Overview 

Using  questionnaires  and  telephone  conversations,  we  searched  for  a  relationship 
between  SEI  maturity  ratings  and  successful  software  development  projects.  In  addition, 
we  evaluated  organizational  acceptance  and  the  impact  of  SEI  assessments  and  reviewed 
organizational  process  improvement  responses  following  assessments.  Our  research 
consisted  of  three  phases  which  will  be  discussed  in  the  following  sections. 

32  Exploratory  Phase 

A  literature  review  of  the  SEI  maturity  model  framework  and  rating  system  as  well 
as  case-study  methodologies  was  performed.  In  addition  to  this  review,  candidate 
software  organizations  that  have  been  evaluated  using  the  SEI  CMM  were  identified  for 
case  study. 

3.3  Research  Design  Phase 

A  case-study  research  design  was  developed  to  accomplish  the  research  goals  within 
accepted  practices  identified  during  the  prior  literature  review.  In  addition,  projects  for 
case-study  were  selected  from  those  identified  in  the  Exploratory  Phase.  Candidate 
organizations  met  the  following  criteria: 

1.  Ai  least  one  prior  SEI  assessment 

2.  Significant  in-house  software  development  or  support 

3.  Willingness  to  participate  in  thesis  research 
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We  used  a  multiple-case  study  design  for  several  reasons: 

1.  Our  research  goals  focus  on  contemporary  events 

2.  Researchers  have  no  control  over  the  respondent  behavi . 

3.  Our  research  questions  are  of  the  how  and  why  form  rather  than  how  much  or 
how  many. 

4.  The  availability  of  quantitative  data  concerning  correlation  between  SEI  ratings 
and  project  success  is  virtually  non-existent 

5.  Case-study  research  can  provide  a  good  pilot  example  for  future  statistically- 
based  efforts. 

According  to  Yin  (20:17), 

"research  methodology  should  be  chosen  based  on  three  conditions: 

(a)  type  of  research  question  posed,  (b)  the  extent  of  control  an 
investigator  has  over  actual  behavioral  events,  and  (c)  the  degree  of  focus 
on  contemporary  as  opposed  to  historical  events." 

According  to  Yin's  matrix  (20:17),  a  case  study  strategy  is  the  most-appropriate  strategy 
for  research  possessing  the  first  three  characteristics  shown  above  as  our  reasons  for 
selecting  the  case-study  approach. 

In  order  to  identify  correlation,  a  multiple-case  study  approach  is  required. 
"Replication,  not  sampling  logic"  (20:48)  is  the  basis  for  multiple- case  studies.  Projects 
(cases)  selected  to  participate  will  be  expected  to  have  either  (a)  similar  results  or  (b) 
contrary  results  but  for  predictable  reasons  (20:49).  While  each  case  will  be  a  free¬ 
standing  study,  the  most  meaningful  analysis  will  be  the  cross-case  comparisons  for 
correlation. 

3.3.1  Data  Gathering.  Study  data  were  gathered  primarily  by  questionnaire 
(Appendix  A).  Clarifications  and  further  information  were  also  gathered  through 
telephone  interviews. 
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The  questionnaire  consisted  of  Software  Engineering  Process  Group  (SEPG)  and 
project  level  questions.  Under  top-level  management's  direction,  SEPG  personnel  define, 
document,  and  introduce  improvements  to  an  organization's  software  process.  Using  the 
SEPG  guidance,  project  level  personnel  develop  and  support  software.  Tne  questionnaire 
was  designed  to  collect  data  at  each  of  these  important  levels  of  software  organizations. 
The  instrument  consisted  of  questions  that  collectively  address  each  of  our  thesis 
objectives.  Once  drafted,  it  was  reviewed  by  advisors  and  software  professionals  from 
organizations  not  selected  to  participate  in  the  case-study.  The  final  questionnaire 
(Appendix  A)  was  sent  to  individuals  from  selected  organizations  and  projects. 

33.2  Data  Analysis.  For  each  organization  under  study,  results  are  presented  as 
they  relate  to  the  thesis  objectives.  Once  the  individual  cases  were  presented,  the  cross¬ 
case  analysis  isolated  common  threads  existing  among  the  cases.  Our  primary  concern 
was  to  determine  if  a  relationship  existed  between  SEI  Capability  Maturity  Model  ratings 
and  project  success.  Reasons  for  organizational  acceptance,  impact,  and  responses  to  SEI 
assessments  were  also  compared  to  address  the  organizational  and  personnel  environment 
responsible  for  the  success  or  failure  of  the  assessments. 

3.4  Research  Execution  Phase 

During  this  phase  we  followed  the  research  plan  set  forth  in  the  previous  section. 

The  process  started  with  case  selection  for  the  multiple-case  study.  Next,  individual 
projects  within  each  case  organization  were  identified.  Data  gathering  included 
assessment  reports,  questionnaires,  and  telephone  conversations.  Finally  the  data  was 
analyzed  with  respect  to  the  research  objectives.  The  following  sections  detail  the  specific 
research  efforts  for  each  of  these  stages. 
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3.4.1  Case  Selection.  The  primary  objective  in  the  case  selection  was  to  include 
DoD  organizations  that  produced  software  in-house.  In  addition,  the  organization  must 
have  been  evaluated  using  the  SErs  Capability  Maturity  Model  at  least  once.  The  second 
requirements  reduced  the  available  pool  of  organizations  considerably  because  relatively 
few  organizations  have  undergone  assessments  to  date.  Finally,  we  also  wanted  to 
compare  similar  types  of  organizations  would  ensure  valid  comparisons  among  the 
individual  cases  studied.  The  Air  Force  Air  Logistics  Centers,  having  recently  been 
assessed  using  the  SEI's  Capability  Maturity  Model,  provided  a  collection  of  organizations 
meeting  all  of  the  requirements  for  case  selection. 

Initially,  five  of  the  Air  Logistics  Centers  (ALCs)  were  contacted.  The  Software 
Engineering  Process  Group  (SEPG)  in  each  was  contacted  via  telephone  and  interviewed. 
During  the  interviews,  each  SEPG  representative  was  asked  about  the  assessment  at  their 
respective  ALC.  The  SEPG  representative  provided  insight  into  several  key  areas  of 
interest  with  respect  to  the  assessment  process,  follow-up  actions,  overall  project 
performance,  and  willingness  to  participate  in  the  study. 

The  interviews  revealed  that  the  ALCs  were  at  different  stages  in  their  process 
improvement  programs.  All  had  been  assessed  at  least  once,  with  one  having  been 
assessed  twice.  Based  on  these  preliminary  interviews,  a  down- selection  to  three  ALCs 
for  case  study  was  made.  The  entire  thesis  team  agreed  that  three  individual  cases 
provided  an  adequate  cross-section,  while  at  the  same  time  limiting  the  study  to  a 
manageable  level  of  effort 

3.42  Data  Gathering. 

3. 4. 2.1  Questionnaire.  In  .ridition  to  SEI  assessment  reports  from  each 
participating  ALC,  the  primary  data  was  collected  through  the  specially  developed 
questionnaire.  The  questionnaire  mirrored  the  research  goals  of  this  project. 


Prior  to  distribution,  the  questionnaire  was  sent  to  several  experienced  software 
professionals,  some  from  non-participating  ALCs,  for  comments  and  validation.  These 
comments  and  suggestions,  together  with  those  of  the  thesis  team  advisors,  were 
incorporated  into  the  final  data  collection  instrument  The  questionnaire  was  subsequently 
distributed  to  the  ALCs  for  completion  by  both  SEPG  and  software  project  personnel. 
Once  again,  each  selected  ALC  was  to  be  considered  as  a  case  which  consisted  of  a 
number  of  individual  projects. 

The  following  three  sections  describe  each  question,  possible  answers,  and  the 
relationships  of  the  question  to  the  research  objectives.  The  discussion  examines  the 
questionnaire  by  research  objective.  To  achieve  our  first  objective,  section  three  of  the 
questionnaire  seeks  to  determine  if  a  relationship  exists  between  project  success  and  SEI 
ratings.  To  fulfill  our  second  objective,  sections  one  and  four  of  the  questionnaire  deal 
with  ALC  acceptance  and  impact  of  the  SEI  assessment.  Finally,  to  accomplish  our  third 
objective,  section  two  of  the  questionnaire  seeks  to  determine  the  effect  on  process 
improvement  resulting  from  the  SEI  assessment  and/or  ongoing  process  improvement 
issues. 


3.42.2  SEI  Maturity  and  Project  Success.  The  first  objective  of  this 
research  is  to  determine  if  a  relationship  exists  between  project  success  and  SEI  capability 
maturity  model  levels.  The  third  section  of  the  questionnaire  concentrates  on  collecting 
data  regarding  this  research  question. 

Question  "3a"  asks  the  respondent  to  identify  the  level  at  which  the  organization  was 
rated  by  the  capability  maturity  model.  Possible  answers  for  this  question  are: 

•  Level  I,  II,  III,  IV,  or  V,  or 

•  Initial,  Repeatable,  Defined,  Managed,  or  Optimizing. 


These  answers  provide  the  first  half  of  the  necessary  information  to  establish  the  existence 
or  non-existence  of  a  relationship. 

Question  "3b’’  asks  what  actions,  taken  as  a  result  of  the  assessment,  have  been  part 
of  the  project.  Answers  to  this  question  suggest  how  the  process  improvement  program 
has  affected  the  project 

Question  "3c''  asks  the  respondent  how  successful  the  subject  project  was  or  is. 
Answers  to  this  question  complete  the  direct  information  required  to  demonstrate  the 
existence  or  non-existence  of  a  relationship  between  CMM  levels  and  project  success.  As 
previously  discussed,  project  success  is  defined  using  the  triad  of  customer  satisfaction, 
costs,  and  schedule  considerations. 

While  questions  "3a,"  "3b,"  and  "3c"  request  data  directly  necessary  to  demonstrate 
the  relationship,  question  "3d"  searches  for  other  possible  reasons  for  improvement.  The 
goal  of  this  question  is  to  determine  if  project  success  can  be  attributed  to  post-assessment 
process  improvements  or  previously  existing  conditions  or  capabilities.  The  question  asks 
how  the  post-assessment  actions  have  affected  the  success  of  the  project  Once  again,  the 
same  criteria  for  project  success  arc  used.  The  purpose  is  to  determine  to  what  extent  the 
assessment  and  resulting  process  improvement  program  have  affected  the  success  of  a 
specific  project 

Finally,  question  "3e"  asks  the  respondent  for  an  opinion  on  whether  the  process 
improvements  will  or  have  directly  benefited  the  project.  In  addition  to  helping  to  identify 
whether  the  process  improvement  program  has  commitment  at  different  levels,  this 
question  clarifies  whether  the  project  success  is  due  to  assessment  related  improvements 
or  business-as-usual. 

3.423  Acceptance  and  Impact  ofSEl  Assessment.  A  second 
objective  of  this  research  is  to  study  the  acceptance  of  the  assessment  process  by 
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management  and  workers  and  the  impact  of  SEI  assessments  on  the  organizations .  The 
primary  reason  for  this  portion  of  the  study  is  to  identify  circumstantial  information 
concerning  the  assessment  and  process  improvements.  As  previously  stated  in  the 
literature  review,  several  authors  have  suggested  that  organizational  commitment  and 
personnel  "buy-in"  are  key  reasons  for  the  success  of  their  process  improvement 
program  (14,  16, 7,  8).  For  these  reasons,  the  first  and  fourth  sections  of  the 
questionnaire  delve  into  the  ALC  environment  prior  to,  during,  and  following  the 
assessment 

Question  "la"  asks  what  factors  led  to  the  initiation  of  a  process  improvement 
program.  Expected  short  answers  will  describe  the  motivation  for  improving  the  software 
process.  Previous  studies  have  followed  similar  patterns  of  grass-roots  interest  in  process 
improvement  to  management  commitment  and  action  (14, 16, 7,  8).  While  the  same  may 
be  true  in  the  Air  Force,  in  some  cases,  organization  factors  have  also  led  to  the 
imposition  of  a  process  improvement  program.  Question  "lb"  continues  with  the 
motivation  theme  by  asking  what  factors  led  to  using  the  SEI  (CMM)  methodology  for 
process  improvement.  Toward  the  same  goal,  question  "le"  asks  for  which  reasons 
striving  for  higher-level  ratings  is  valuable. 

Questions  "lc"  and  "Id"  identify  the  respondent  s  general  opinion  of  SEI,  CMM,  and 
assessment  methodologies.  Question  "lc"  asks  for  an  overall  opinion  of  the  SEI 
assessment  process/methodology.  Short  responses  will  identify  top-level  advantages  and 
disadvantages  of  the  SEI  methodology.  Finally,  question  "Id"  asks  respondents  whether 
tiie  maturity  level  is  a  legitimate  representation  of  an  organizations  ability.  The  purpose  of 
this  question  is  to  address  a  prior  critique  of  the  SEI,  that  the  maturity  level  number  was 
too  simple  to  characterize  the  complex  software  process  (3). 

Where  section  one  concerned  motivation  and  management  level  motivation  for  the 
process  improvement  program,  section  four  concerns  the  personnel  and  management 
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reaction  to  the  program.  Question  "4a"  asks  for  the  worker  level  feeling  concerning  the 
SEI  assessment  Parallel  to  management  commitment,  this  personnel  "buy-in"  has  been 
identified  as  a  key  reason  for  process  improvement  success  (14,  16,  7,  8).  Question  "4b" 
asks  for  management's  reaction  to  the  assessment.  This  question's  purpose  is  to  see  if 
motivation  for  process  improvement  corresponds  to  reaction.  One  possible  example  could 
be  that  process  improvement  was  directed  by  higher  authorities  and  ALC  management 
actively  pursues  the  resulting  process  improvement  initiatives.  Question  "4c"  concerns 
management's  pursuit  of  the  assessment  recommendations. 

Question  "4d"  asks  for  any  negative  effects  that  resulted  from  the  SEI  assessment 
Similarly  Question  "4e"  examines  whether  the  desired  non-attributive  environment  has 
been  maintained  or  if  administrative  discipline  has  accompanied  the  process.  Both 
questions  concern  how  completely  the  workers  and  management  have  embraced  the  SEI 
methodology.  Finally  question  "4f '  asks  whether  the  environment  has  changed  since  the 
assessment 

3. 4. 2. 4  Process  Improvement  Responses.  The  final  research  objective 
is  to  study  organizational  process  improvement  responses  following  assessments.  Section 
two  in  the  questionnaire  explores  this  area.  In  order  to  determine  to  what  extent  the 
assessment  recommendations  came  from  within  the  organization,  question  "2a"  asks 
whether  the  assessment  included  SEI  personnel  only,  local  personnel  only,  or  a 
combination  of  the  two. 

Questions  "2b",  "2c’  and  "2d"  directly  investigate  what  actions  have  been 
taken  as  a  result  of  the  assessment,  what  other  unrelated  process  actions  have  taken  place, 
and  what  the  benefits  have  been  realized  through  the  SEI  assessment  process.  The  final 
question  "2e"  asks  what  process  improvements  have  been  identified.  Expected  answers 
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should  include  only  top-level  issues  such  as  training  and  configuration  management 
These  answers  help  put  the  corrective  actions  in  context. 

3.4.3  Data  Analysis 

For  each  case  (aLC)  considered,  a  history  was  developed  detailing  the 
assessment(s),  organizational  acceptance  of  the  assessment(s),  implementation  of  process 
improvements  resulting  from  the  assessment,  and  the  effect  on  projects  considered  within 
each  ALC.  The  primary  source  of  information  was  the  questionnaire  responses;  however, 
follow-up  interviews  were  used  to  provide  additional  information  where  necessary.  Once 
the  individual  cases  had  been  analyzed,  a  cross-case  analysis  was  performed. 

The  cross-case  analysis  consisted  of  analyzing  the  three  individual  cases  as  a  whole, 
rather  than  separately.  Of  primary  interest  was  to  identify  common  factors  contributing  to 
project  success  or  any  of  the  other  factors  in  the  individual  case  analysis.  Based  on  the 
cross  case  analysis,  inferences  were  made  as  to  certain  conditions  or  activities  that  had 
contributed  to  the  overall  success  or  failure  of  individual  projects  and  the  assessments 
>  nselvcs. 

3.5  Summary 

The  objectives  of  this  thesis  were  three-fold:  (1)  determine  if  a  relationship  exists 
between  SEl  Capability  Maturity  Model  ratings  and  project  success;  (2)  evaluate 
organizational  acceptance  and  impact  of  SEI  assessments  for  the  purpose  of  establishing 
whether  the  environment  was  conducive  to  an  effective  process  improvement  program; 

(3)  review  organizational  process  improvement  responses  following  assessments  because  a 

H 

^process  improvement  program  is  the  mechanism  for  achieving  improved  quality,  lower 
cost,  and  on-time  software  projects.  These  objectives  were  met  in  a  three-phased  manner. 
The  exploratory  phase  consisted  of  literature  review  and  project  selection.  The  data 
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gathering  phase  consisted  of  telephone  conversations  and  questionnaires.  Finally,  the 
analysis  phase  included  single-case  study  summaries  and  cross-case  comparisons. 
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IV.  Individual  Case  Analysis 


4.1  Background  Behind  the  Air  Logistics  Center  Assessments 

Because  the  ALCs  and  other  DoD  maintenance  organizations  roust  compete  for  a 
share  of  DoD  resources,  Air  Force  Materiel  Command  (AFMC)  required  that  the  Air 
Logistics  Centers'  (ALCs)  software  production/maintenance  process  be  assessed  via  the 
SEI CMM  The  goal  of  the  assessment  program  is  continuous  process/quality 
improvement,  res  ilting  in  more  effective  maintenance  organizations  better  able  to  compete 
in  Jie  DoD  market  The  assessment  program  is  managed  by  the  Embedded  Computer 
Resources  Office  at  Air  Force  Material  Command  (HQ  AThfC/ENSR).  All  five  ALCs 
have  been  assessed  with  four  achieving  a  Level  I  maturity  rating. 

AT vlC  plans  to  have  erxh  ALC  assessed  every  two  to  three  years  with  the  second 
round  of  SEI  assessments  to  be  completed  by  the  end  of  1994.  The  goal  ol  AFMC  is  for 
each  ALC  to  achieve  a  Level  III  rating  by  1998.  This  goal  is  consistent  with  the  Air  Force 
guidance  that  all  Air  Force  softwrre  producir.g/maintaining  organizations  to  achieve  a 
maturity  rating  of  Level  HI  by  1998. 

4.2  Case  Study  Ground  Rules 

Because  the  ALCs  have  requested  confidentiality,  references  to  specific  documents 
and  ALCs  will  not  be  made.  It  should  be  understood  that  a  significant  portion  of  the 
information  contained  in  the  case  analyses  was  obtained  from  internal  planning  documents, 
assessment  findings  reports,  and  questionnaires  that  are  not  releasable.  References  to  the 
ALCs  and  individual  projects  have  been  eliminated  to  maintain  confidentiality  and  conceal 
the  identities  of  the  organizations  and  individuals  participating  in  the  case  studies. 
Responses  to  the  questionnaires,  without  reference  to  individuals  or  organizations,  have 
been  tabulated  in  appendices  B  through  D. 


4.3  ALC  Number  1 


In  the  late  1980  s,  ALC  Number  1  desired  to  implement  a  software  process 
improvement  program.  Management  was  introduced  to  the  SEI  CMM  in  1989  and 
believed  that  it  could  be  used  to  assist  them  in  assessing  their  process  and  implementing  a 
process  improvement  program.  This  interest  in  the  SEI  CMM  resulted  in  the  beginning  of 
serious  software  process  improvement  efforts  at  ALC  Number  1. 

ALC  Number  1  was  first  assessed  in  March  1990  by  a  team  consisting  of  SEI  and 
ALC  personnel  who  had  completed  the  SEI  assessment  training  program.  The 
assessment  was  viewed  rather  skeptically  by  a  large  number  of  ALC  software  personnel 
but  was  supported  strongly  by  management.  At  this  time,  the  ALC  was  found  to  be  at 
CMM  Level  I,  the  initial  level.  The  1990  self-assessment  provided  insight  into  the 
software  process  at  ALC  Number  1,  and  the  results  were  used  to  formulate  an  initial 
Action  Plan  which  was  the  beginning  of  the  structured  software  process  improvement 
program. 

Specifically  as  a  result  of  this  first  assessment,  two  permanent  teams  were 
established  to  guide  process  improvement  efforts.  The  first  was  a  steering  committee  that 
has  been  meeting  monthly  to  discuss  and  quantify  process  improvemer>f  '»rts.  The  other 
was  a  software  engineering  process  group  (SEPG)  that  was  formed  to  On  and 
implement  a  process  improvement  program.  Most  of  the  effort  since  the  first  assessment 
has  been  spent  putting  together  the  process  improvement  infrastructure. 

ALC  Number  1  was  again  assessed  in  March  1993  by  a  team  of  ALC  and  SEI 
personnel.  This  assessment  was  the  first  performed  using  the  SEI  CMM  Version  1.1,  the 
^February  1993  version.  The  format  for  this  assessment  was  similar  to  the  first  assessment, 
involving  a  similar  number  of  projects  and  personnel.  According  to  interviews  with  an 
SEPG  representative,  the  second  assessment  was  supported  much  more  strongly  than  the 
first,  largely  due  to  the  continuous  focus  on  process  improvements  during  the  previous 


three  years.  Efforts  since  the  first  assessment,  having  been  focused  on  developing  the 
process  improvement  infrastructure,  showed  that  process  improvement  coulc  be 
accomplished  in  a  structured,  common-sense  manner.  Employee  acceptance  o.  ihe 
ongoing  process  improvement  concept  was  enhanced  as  a  result  of  these  efforts.  ALC 
Number  1  was  found  to  be  at  Level  D  as  a  result  of  this  assessment. 

There  were  forty-four  specific  improvements  implemented  between  the  March  1990 
and  March  1993  assessments.  Process  improvement  efforts  have  largely  been  goal 
oriented,  focusing  on  employee  morale,  customer  satisfaction,  and  product  quality. 
Continuous,  non-intrusive,  measurement  of  these  three  aspects  are  made  through  monthly 
meetings  with  supervisors  and  the  SEPG.  The  goal  is  not  to  constantly  monitor  the  low- 
level  effects  of  process  improvement  efforts,  but  to  observe  the  effects  on  the  three 
aspects  referred  to  above.  Throughout  the  process  improvement  effort,  begun  in  1990, 
management  has  been  extremely  supportive.  ALC  Number  1  has  a  goal  of  achieving  a 
Level  III  assessment  rating  by  1995,  well  in  advance  of  AFMCs  goal  of  Level  Ill  by  1998. 

The  following  two  sections  address  the  assessment  findings  and  the  process 
improvement  program  at  ALC  Number  1 .  These  two  sections  are  the  product  of  our  data 
collection  efforts  which  include  questionnaires,  interviews,  and  the  findings  and  follow-up 
action  reports  from  ALC  Number  1.  The  third  section  analyzes  the  data  with  respect  to 
our  three  objectives. 

4.3.1  Summary  of  Assessment  Findings.  The  findings  of  the  March  1993 
assessment  showed  that  there  are  six  key  areas  which  need  to  be  addressed  in  the 
continuing  process  improvement  effort.  Each  area  is  defined  by  first  explaining  the  key 
area  and  citing  the  specific  problem(s)  associated  with  it 

1 .  Organized  Process  Definition  (Organizational  process  definition  involves 
developing  and  maintaining  the  organization's  standard  software  process  along 
with  related  process  assets.) 
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Specific  Problem:  There  is  no  single  software  process  architecture.  Little 
guidance  is  provided  on  how  software  process  guides  are  to  be  tailored  to 
create  individual  project  processes  or  how  the  tailoring  is  to  be  documented 
and  approved. 

2.  Training  Program  (The  training  program  involves  first  identifying  the 
training  needed  by  the  organization  projects,  and  individuals,  then  developing 
or  procuring  training  to  address  the  identified  needs.) 

Specific  Problem:  Training  needs  are  not  systematically  identified  and  used 
to  plan  future  training.  For  some  roles  such  as  project  leader,  training  is  not 
required  by  the  organization  or  routinely  provided. 

3.  Integrated  Software  Management  (Integrated  software  management 
involves  developing  the  project's  defined  software  process  and  managing  the 
software  project  using  this  defined  software  process.) 

Specific  Problem:  Lessons  learned  from  project  process  execution  are  not 
routinely  collected  and  made  available  to  all  personnel. 

4.  Software  Product  Engineering  (Software  product  engineering  involves 
performing  the  engineering  tasks  to  build  and  maintain  the  software  using  the 
defined  software  process  and  appropriate  methods  and  tools.) 

Specific  Problem:  No  specific  problem  areas  noted,  but  this  area  was 
identified  as  an  area  of  concern  to  be  monitored  as  process  improvement 
continues.  This  area  is  closely  related  to  other  key  process  areas. 

5.  Peer  Reviews  (Peer  reviews  involve  a  methodical  examination  of  software 
word  products  by  the  producers'  peers  to  identify  defects  and  areas  where 
changes  are  needed.) 

Specific  Problem:  Peer  reviews  are  not  planned  and  conducted  in  a 
consistent  manner  across  the  organization.  There  are  no  documented 
procedures  for  conducting  peer  reviews.  PeeT  review  leaders  are  not  formally 
trained  and  measurements  on  peer  reviews  are  not  collected  and  analyzed. 

6.  Coordination  with  the  Software  Control  Center  (Does  not  directly 
relate  to  the  CMM  but  is  recognized  as  an  issue  that  must  be  considered 
because  the  Software  Control  Center  is  believed  to  be  a  very  reliable  source  of 
strong  configuration  management  support.) 
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Specific  Problem:  Contractors  do  not  always  follow  the  proper  Computer 
Program  Identification  Number  (CPIN)  request  procedures.  Turn-around  time 
through  the  Software  Control  Center  is  highly  variable. 

432  Process  Improvement  Program.  ALC  Number  1  has  updated  their 
Action  Plan  to  include  the  findings  of  the  March  1993  assessment.  This  updated  Action 
Plan  is  the  mechanism  by  which  process  improvement  efforts  needed  to  resolve  the 
specific  problems  identified  in  the  March  1993  assessment  can  be  addressed.  The  Action 
Plan  also  includes  other  improvements  not  directly  related  to  assessment  findings. 

In  orde~  to  facilitate  process  improvement,  ALC  Number  1  employs  two  permanent 
groups  to  manage  the  implementation  of  the  process  improvement  program.  These 
groups  were  formed  as  a  result  of  the  March  1990  assessment  The  first  is  a  management 
steering  team  that  provides  management  support  for  the  process  improvement  program 
and  is  composed  of  division  and  branch  level  management  The  second  is  the  SEPG.  The 
SEPG  provides  the  technical  implementation  details  of  how  best  to  implement  specific 
improvements. 

In  addition  to  the  two  permanent  groups,  ad  hoc  technical  working  groups  (TWGs) 
are  formed  as  needed  in  response  to  specific  problems  in  specific  areas.  The  SEPG 
coordinates  the  activities  of  the  management  steering  group  and  the  TWGs  and  maintains 
the  overall  process  improvement  plan. 

Once  a  specific  improvement  is  approved  by  the  management  steering  team,  a  TWG 
is  usually  established  to  work  the  initiative.  The  TWG  drafts  a  Tactical  Plan  that  includes 
a  statement  of  why  the  organization  is  adopting  a  new  technology  or  procedure.  In 
addition,  the  Tactical  Plan  notes  the  goals  of  the  new  initiative  and  an  estimate  of  the 
return  on  investment  for  the  project.  Next,  the  problem  and  possible  solutions  are 
researched,  a  solution  is  proposed,  and  an  implementation  plan  for  the  solution  is 
developed. 
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A  prototype  group  is  now  chosen  to  test  the  new  procedure  or  technology,  and 
measures  and  indicators  are  decided  upon  to  monitor  the  impact  of  the  change.  Measures 
are  taken  prior  to  the  implementation  to  establish  a  baseline  against  which  to  compare 
measures  taken  afterward.  The  implementation  is  monitored  for  some  time  (prototype 
period  varies  with  specific  improvement)  to  determine  if  it  is  successful 

If  a  process  improvement  has  succeeded  in  the  prototype  phase,  it  is  implemented 
throughout  the  organization.  The  improvement  is  subject  to  the  same  measures  and 
indicators  after  broad  implementation  to  monitor  its  status.  The  improvement  is 
monitored  closely  until  it  has  been  determined  to  be  successful  at  which  time  the 
improvement  initiative  is  closed,  and  it  is  accepted  as  a  part  of  the  software  process. 

The  success  of  improvement  efforts  is  measured  in  two  ways.  The  first  is  an 
employee  satisfaction  survey  that  solicits  suggestions  and  feedback  on  the  process 
improvement  program.  The  second  is  the  SEI CMM.  The  next  assessment  is  planned  for 
late  1995. 

Because  ALC  Number  1  was  assessed  most  recently  in  March  1993,  there  are  no 
specific  examples  of  improvements  that  have  been  implemented  directly  as  a  result  of  these 
findings.  However,  this  mechanism  was  used  to  advance  ALC  Number  1  from  a  Level  I 
organization  (March  1990)  to  a  Level  II  organization  (March  1993),  and  according  to 
interviews  with  the  SEPG  leader,  it  has  been  very  effective  and  is  well  accepted  by  ALC 
personnel 

Finally,  interviews  with  the  SEPG  revealed  that  improvements  are  not  implemented 
directly  in  response  to  SEI  assessment  findings,  but  arc  implemented  to  fix  perceived 
problems  that  may  only  indirectly  effect  the  maturity  level  Tins  method  of  identifying  and 
correcting  problems,  rather  than  responding  to  a  template  for  improvement  has  allowed 
ALC  Number  1  to  take  ownership  for  the  improvement  program  resulting  in  well 
respected  and  accepted  program. 
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433  Analysis  for  a  Relationship  between  Project  Success  and  SEJ 
CMM  Ratings.  Our  first  objective  was  to  determine  if  a  relationship  existed  between 
SEI  Capability  Maturity  Model  ratings  and  project  success.  Unfortunately,  because  the 
most  recent  assessment  was  in  March  1993,  information  from  the  questionnaires  and 
interviews  provided  little  data  on  it's  impact  on  project  success,  although  one  of  the 
projects  commented  that  it  anticipated  positive  effects  on  project  success  due  to  the 
continuing  process  improvement  efforts. 

43.4  Analysis  of  Organizational  Acceptance  and  Impact.  The  second 
objective  was  to  evaluate  the  organizational  acceptance  and  impact  of  the  SEI  assessment 
process.  One  of  the  SEPG  respondents  stated  that  some  fust-level  supervisors  were 
"somewhat  resistant"  to  the  assessments  because  the  improvements  were  perceived  as 
taking  too  much  time  for  the  potential  benefit.  However,  this  opinion  was  not  typical  of 
all  supervisors.  In  addition,  all  three  SEPG  respondents  indicated  that  management  has 
definitely  maintained  the  non-attributive  environment  within  the  ALC.  The  preservation 
of  a  non-attributive  environment  enhances  the  effectiveness  of  the  assessment  by 
encouraging  more  candid  responses,  resulting  in  more  accurate  assessment  findings.  Two 
of  the  three  SEPG  responses  stated  that  there  were  no  negative  effects  as  a  result  of  the 
assessment.  The  remaining  respondent  did  not  indicate  that  there  were  any  negative 
effects. 

The  SEPG  responses  to  the  first  section  of  the  questionnaire  indicated  that  the 
results  of  the  assessment  were  valid  and  represented  the  actual  state  of  the  software 
process  at  ALC  Number  1.  Two  of  the  three  respondents  from  the  SEPG  stated  that  it 
was  good  to  have  an  outside  organization  assess  their  process  and  identify  areas  of 
weakness.  The  assessment  results  were  the  basis  for  the  Action  Plan  which  identifies 
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specific  improvements  to  be  implemented  to  meet  the  address  specific  problems  in  key 
process  areas. 

Responses  from  both  of  the  projects  noted  that  the  worker  level  reaction  was 
positive.  Both  projects  also  indicated  that  they  benefited  by  being  made  aware  of  their 
strengths  and  weaknesses.  Both  projects  stated  that  they  believed  that  management  was 
pleased  with  the  results  of  the  assessment.  Both  projects  indicated  that  there  were  no 
negative  reactions  due  to  the  assessments  and  that  the  non-attributive  environment  was 
maintained. 

435  Analysis  of  Organizational  Process  Improvement  Responses.  The 
third  research  objective  was  to  review  organizational  process  improvement  following  the 
assessments  because  the  process  improvement  program  is  the  mechanism  for  achieving 
improved  quality,  lower  cost,  and  on-time  software  projects.  The  focus  at  ALC  Number  1 
has  not  been  to  respond  to  assessment  results  and  implement  improvements  specifically  to 
fix  problems.  ALC  Number  1  has  spent  considerable  effort  developing  the  process 
improvement  infrastructure  detailed  in  the  section  on  the  process  improvement  program. 
Interviews  with  the  SEPG  Leader  confirmed  that  this  infrastructure  is  used  as  described  to 
implement  and  measure  process  improvements.  Also,  as  previously  stated,  forty-four 
specific  improvements  had  been  implemented  between  the  1990  and  1993  assessments. 

The  questionnaire  responses  stated  that  several  process  improvement  initiatives  were 
begun  at  ALC  Number  1  as  a  result  of  the  most  recent  (March  1993)  assessment  One  of 
the  projects  stated  that  peei  reviews  were  begun.  In  addition,  the  other  project  is  "in  the 
process  of  developing  better  and  documentable  metrics  for  our  projects."  One  of  the 
projects  also  stated  that  the  identification  of  training  needs  as  well  as  monthly  meetings  to 
address  a  standardization  of  the  software  process  were  begun  as  a  result  of  the 


assessment  These  improvement  efforts  directly  address  the  findings  of  the  most  recent 
assessment 

Following  the  March  1993  assessment  the  Action  Plan  was  completed  by  mid-July 
1993  using  the  assessment  recommendations.  This  rapid  turnaround  suggests  that  ALC 
Number  1  respects  the  findings  and  is  very  much  concerned  with  continuing  process 
improvement 

Section  two  of  the  questionnaire  deals  with  the  process  improvement  issue  relating 
the  assessments.  One  SFPO  respondent  noted  that  on-going  process  improvements  were 
re-focused  as  a  result  of  *tc  itqent  assessment,  again  illustrating  the  attention  paid  to  the 
assessments  and  process  improvement  in  general.  The  overall  status  of  the  process 
improvement  prognm  at  ALC  Number  1  is  such  mat  continuing  improvement  is  the  nonn 
rather  than  the  exception.  The .  trueture  that  has  been  established  seems  to  be  effective  at 
implementing,  maiv'-guig,  and  measuring  process  improve trvtnt  effoi  ts. 

As  a  result  of  interviews  with  tltc  SEK3  leader,  it  is  clear  that  throughout  Lie 
process  improvement  effort,  begun  in  19f0,  management  has  been  extremely  supportive. 
The  assessments  have  been  the  means  by  which  ALC  Number  1  assesses  the  overall 
maturity  status  of  their  software  process.  Their  focus  on  process  improvement, 
specifically  their  commitment  to  a  well  defined  process  improvement  plan,  has  resulted  in 
an  effective  continuing  process  improvement  environment. 

The  successes  of  ALC  Number  1  are  not  due  to  a  blind  acceptance  of  the  CMM,  but 
to  implementation  of  those  improvements  that  further  those  goals  which  advance  the 
effectiveness  of  the  software  process.  By  following  their  action  plan,  which  was 
formulated  with  the  assessment  findings  in  mind,  ALC  Number  1  will  address  each  key 
process  area  that  was  identified  as  being  deficient.  ALC  Number  1  has  a  goal  of  achieving 
a  Level  III  assessment  rating  by  1995,  well  in  advance  of  AFMC’s  goal  of  Level  BI  by 
1998. 
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4.4  ALC  Number  2 


ALC  Number  2  was  assessed  in  March  1992  by  a  team  of  personnel  from  the  SEI, 
SEPG,  and  from  other  ALCs.  The  rationale  for  the  SEI  process  assessment  and 
subsequent  improvement  program  is  two-fold.  First,  Air  Force  directives  mandate  in- 
house  software  developers  achieve  a  Level  HI  rating  by  1998.  Second,  ALC  management 
sought  to  improve  their  software  capability  in  order  to  be  more  competitive  for  future 
software  development  and  support  work.  One  of  the  process  related  problems  identified 
before  the  assessment  was  a  lack  of  management  visibility  into  the  software  process.  The 
result  of  this  problem  was  that  management  did  not  become  aware  of  problems  until  the 
problems  had  become  catastrophic. 

During  the  assessment,  leaders  of  five  projects  completed  the  SEI  questionnaire  and 
were  interviewed  by  the  assessment  team  to  clarify  their  answers  and  address  specific 
concerns.  The  assessment  concluded  with  a  briefing  to  the  ALC  Commander,  his  staff, 
and  most  of  the  software  development  personnel. 

A  thorough  report  on  the  assessment  findings  and  recommendations  was  created  and 
distributed  shortly  after  the  assessment  detailing  the  results  and  process  improvements 
necessary  to  meet  identified  areas  of  concern.  This  report  was  used  as  the  basis  for 
developing  a  specific  action  plan  which  will  help  guide  the  ALC  to  their  goal  -  a  Level  HI 
rating  by  March  of  1994. 

Sources  for  information  for  this  case  study  include  questionnaires,  the  assessment 
report,  and  telephone  conversations  with  ALC  personnel.  Questionnaires  were  sent  to 
software  engineering  process  group  (SEPG)  personnel  as  well  as  key  software  personnel 
from  several  representative  projects  within  the  software  engineering  division  of  ALC 
Number  3.  A  summary  of  the  questionnaire  responses  for  ALC  Number  3  is  included  as 
Appendix  D.  The  next  section,  summary  of  assessment  findings,  reviews  the  process 
assessment  report  After  the  assessment  report  discussion,  the  process  improvement 
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program  section  outlines  the  ALC's  subsequent  process  improvement  efforts.  Telephone 
conversations  with  SEPG  leaders  and  members  were  the  sources  for  process  improvement 
information. 

4.4.1  Summary  of  Assessment  Findings.  The  assessment  resulted  in  ALC 
Number  2  receiving  a  Level  I  maturity  rating.  Seven  areas  of  concern  were  noted. 
Improvements  in  the  first  five  are  needed  to  advance  to  a  Level  II  maturity  rating,  while 
the  last  two  areas  require  improvement  to  develop  an  infrastructure  in  which  process 
improvements  can  be  implemented  and  sustained. 


1.  Requirements  Management  (Involves  establishing  and  maintaining  an 
understanding  with  the  customer  on  the  requirements  for  the  software 
throughout  the  life  cycle.) 

Specific  Problem:  Project  requirements  are  not  always  clearly  stated. 

2.  Project  Planning  (Involves  project  planning  at  the  developing  organization 
and  uses  the  previously  developed  requirements  specification  to  estimate 
required  resources  and  guides  the  process  of  meeting  customer  requirements) 

Specific  Problem:  Project  planning  and  resource  estimating  are  not  always 
effective.  Ineffective  process  for  documenting  project  plans  and  estimates. 

3.  Project  Management  and  Oversight  (Involves  tracking  and  reviewing  the 
software  accomplishments  and  results  against  documented  estimates, 
commitments,  and  plans.) 

Specific  Problem:  Perception  exists  that  management  is  not  involved,  does 
not  understand,  and  is  not  concerned  about  software  issues.  Project  status  not 
adequately  tracked.  Perception  exists  that  people  are  working  on  unrelated 
activities. 

4.  Configuration  Management  (Involves  identifying  configuration  of  a 
system  at  discrete  points  in  time  and  maintaining  the  integrity  of  a  the  system 
as  it  is  modified.) 
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Specific  Problem:  There  is  a  lack  of  a  basic  understanding  of  the 
configuration  management  function.  Lack  of  a  documented  configuration 
management  process. 

5.  Software  Quality  Assurance  (Involves  reviewing  and  auditing  software 
products  and  activities  to  ensure  that  they  comply  with  applicable  product  and 
process  standards.) 

Specific  Problem:  Lack  of  standards  guidelines  or  procedures  for  the 
software  life  cycle.  Software  quality  functions  under  total  quality  management 
(TQM)  concept  are  not  clearly  defined  or  understood. 

6.  Acquisition  (Involves  both  the  supponability  by  the  ALC  of  a  system 
following  SPO  acquisition,  and  the  procurement  of  tools  and  materiel  needed 
for  internal  work.) 

Specific  Problem:  Support  issues  not  adequately  addressed.  Local 
procurement  of  parts  and  tools  inefficient  resulting  in  schedule  slippage. 

7.  Human  Resources  (Involves  the  training  of  personnel  within  the  ALC.) 

Specific  Problem:  Inadequate  software  training  in  project  and  software 
management  as  well  as  other  related  areas.  Inadequate  emphasis  on  personnel 
issues  such  as  career  progression. 

The  summary  of  assessment  findings  will  be  the  basis  for  discussion  in  the  next 
section  where  individual  questionnaire  responses  will  be  addressed 


4.4.2  Process  Improvement  Program.  As  stated  earlier,  the  final  assessment 
report  was  published  in  December  1992.  Since  that  time,  the  SEPG  has  been  involved 
drafting  the  action  plan  and  addressing  the  first  three  unsatisfied  Level  II  Key  Process 
Areas  (KPAs)  identified  in  the  assessment  report.  According  to  the  SEPG  chief,  project 
planning  and  project  management  and  oversight  were  two  of  these  high  priority  KPAs  that 
directly  relate  to  realistic  project  expectations  and  increasing  management  visibility  into 
the  software  process.  The  third  KPA  of  immediate  concern  was  requirements  definition. 
Improvement  in  defining  requirements  is  expected  both  to  increase  customer  satisfaction 
and  reduce  long-term  costs. 
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The  SEPG  chief  stated  that  with  these  three  efforts  underway,  teams  are  being 
formed  to  address  software  quality  assurance  and  software  configuration  management 
These  two  areas  represent  the  last  KPAs  required  for  ALC  Number  2  to  achieve  a  Level  D 
rating. 

Although  not  CMM  KPAs,  two  other  efforts  are  planned  once  the  five  KPA  teams 
are  formed  and  working.  These  efforts  are  intended  to  improve  the  acquisition  and  human 
resources  areas  of  the  ALC.  The  effort  in  acquisition  must  concentrate  on  establishing  a 
timely  and  flexible  means  to  acquire  tools  needed  to  compete  for  and  accomplish  software 
efforts.  The  second  category  for  improvement  was  human  resources,  where  lack  of 
technical  and  managerial  training,  lack  of  a  technical  career  path,  ineffective  rewards  for 
performers,  and  failure  to  challenge  non-performing  team  members  are  major  deficiencies. 

A  great  deal  of  time  is  required  to  begin  efforts  in  these  seven  process  categories  and 
even  more  time  will  be  required  to  demonstrate  improvements.  For  these  reasons,  the 
SEPG  estimates  that  the  organization  will  now  achieve  their  Level  II  goal  in  September  of 
1994  rather  than  March  of  1994. 

4.43  Analysis  for  a  Relationship  between  Project  Success  and  SEI 
CMM  Ratings.  Our  first  objective  was  to  determine  if  a  relationship  exists  between  SEI 
Capability  Maturity  Model  ratings  and  project  success.  Section  three  of  the  questionnaire 
and  subsequent  follow-up  telephone  conversations  were  the  sources  of  information  for  this 
objective. 

Prior  to  the  assessment,  management  did  not  have  visibility  into  the  software 
process.  Because  management  was  not  involved  in  the  process,  they  were  unable  to  see 
^problems  growing  and  were  unable  to  estimate  costs  and  schedules.  Among  the 

assessment  findings  were  shortfalls  in  project  planning,  project  management  and  oversight. 


and  requirements  definition.  In  response,  teams  were  set  up  to  implement  improvements 
in  these  areas. 

Both  project  respondents  from  ALC  Number  2  indicated  that  their  projects  were 
already  more  successful  than  before  the  assessment,  in  terms  of  quality  and  customer 
satisfaction.  Much  of  the  credit  for  this  success  has  been  attributed  to  the  early  process 
improvements  of  increased  customer  interaction,  cost  and  schedule  tracking,  and  some 
process  standardization.  Each  respondent  indicated  that  the  ability  to  accurately  predict 
costs  and  schedules  has  impressed  customers.  In  this  case,  improvement  within  the  Level 
I  rating  has  already  led  to  increased  project  success,  specifically  regarding  the  quality  and 
customer  satisfactionaspects  of  project  success.  This  relationship  will  also  be  studied 
across  all  three  ALCs  in  the  Cross-Case  Analysis  (Chapter  V). 

4.4.4  Analysis  of  Organizational  Acceptance  and  Impact.  The  second 
objective  was  to  evaluate  the  organizational  acceptance  and  impact  of  the  SEI  assessment 
process.  Sections  one  and  four  of  the  questionnaire  and  the  assessment  report  are  the 
primary  sources  for  information  related  to  this  objective. 

ALC  Number  2's  primary  motivation  for  the  SEI  process  assessment/improvement 
program  was  the  Air  Force  mandate  to  achieve  Level  III  by  1998.  SEPG  members 
understand  the  CMM  and  its  associated  advantages  and  disadvantages.  ALC  Number  2 
adapted  their  process  improvement  effort  to  include  human  resources  issues,  which  are  not 
specifically  addressed  by  the  CMM.  One  SEPG  member  stated  that  customers  are 
concerned  only  about  cost,  schedule,  and  quality  --  not  SEI  levels.  Other  respondents, 
however,  agreed  that  the  maturity  level  is  a  valid  measurement  of  their  software 
generation  and  support  capability. 

The  worker  level  response  ranged  from  "wait  and  see"  to  active  support.  The  two 
SEPG  respondents  listed  the  relatively  slow  rate  of  change  and  a  lack  of  quick  and  visible 
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results  as  reasons  for  apprehension  among  workers.  One  project,  respondent  noted  that 
the  worker  level  acceptance  appeared  to  slowly  improve  with  time.  The  second  project 
respondent  stated  that  workers  felt  that  the  assessment  and  improvement  program  were 
"just  another  formality." 

Opinions  of  management's  response,  again,  ranged  from  solid  acceptance  to  mere 
compliance  with  the  An  Force  mandate.  SEPG  members  stated  that  management  was 
initially  just  "filling  me  square,"  but  had  recently  begun  to  back  the  program.  At  the 
project  level,  one  respondent  stated  that  management  had  to  be  forced  to  participate;  the 
othr  management's  interest  had  slacked  off  from  its  initial  backing.  These 

differ.  ’pinions  could  be  attributed  to  workers'  different  points  of  reference  with 

respect  to  management  Workers  deal  with  project  leaders,  branch  chiefs,  and  section 
chiefs.  SEPG  members  work  with  all  levels  of  management  within  the  ALC.  In  all  cases, 
respondents  stated  that  management  had  maintained  the  non-attributive  environment 

According  to  all  respondents,  the  software  professionals  of  ALC  Number  2  have 
cautiously  accepted  the  SEI  assessment  and  improvement  efforts.  Since  the  process 
assessment,  support  has  grown.  Management’s  reaction  to  the  SEI  program,  according  to 
respondents,  spanned  the  range  from  forced  acceptance  to  active  interest.  SEPG  members 
felt  slightly  more  positive  about  management’s  reaction  than  did  project  personnel.  In 
addition,  SEPG  members  indicated  that  management  support  for  the  process  improvement 
effort  was  growing,  while  project  respondents  stated  that  managerial  efforts  had  slacked 
off.  Despite  these  disagreements,  project  level  workers  indicated  direct  process 
improvements  that  involve  managers  of  their  projects. 

Both  project  respondents  agree  that  the  improvenjent  programs  have  already 
affected  their  projects  through  increased  quality  and  customer  satisfaction  and  more 
accurate  cost  and  schedule  estimation.  While  improvements  are  not  currently  quantifiable, 
the  cost  and  schedule  tracking  information  will  enable  future  return  on  investment  (ROI) 
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calculations.  The  ALC  will  use  the  ROI  to  gauge  the  value  of  the  improvement  program. 
All  respondents  expect  further  cost  and  schedule  improvements. 

4.4.5  Analysis  of  Organizational  Process  Improvement  Responses.  The 
third  research  objective  is  to  review  organizational  process  improvement  responses 
following  assessments.  Section  two  of  the  questionnaire  and  the  telephone  conversations 
with  SEPG  members  are  the  sources  of  information  for  this  discussion. 

ALC  Number  2  initially  responded  to  the  assessment  with  a  very  thorough  findings 
and  recommendations  report.  Process  groups  were  created  to  address  requirements 
definition,  project  planning,  and  project  management  and  oversight  As  previously  stated, 
project  personnel  have  already  realized  quality  and  customer  satisfaction  improvements 
that  have  directly  resulted  from  the  efforts  of  die  these  process  groups.  In  addition, 
management  and  executive  level  committees  were  established  to  guide  software 
engineering  process  improvement  These  groups  have  increased  management  awareness 
and  involvement  in  the  improvement  effort 

The  SEPG  is  currently  drafting  specific  strategic  and  tactical  action  plans  to  help  the 
software  engineering  division  achieve  its  goal  of  Level  II  by  March  of  1994.  Due  to  the 
slower  than  expected  pace  of  change,  SEPG  members  now  expect  to  achieve  Level  n  by 
September  of  1994.  Process  groups  have  recently  been  established  to  address  the 
software  configuration  management  and  software  quality  assurance.  Achievement  of  the 
goals  of  these  two  key  process  areas,  plus  the  three  underway,  are  the  only  remaining 
KPAs  required  for  a  Level  II  rating.  Once  these  efforts  are  on  track,  the  SEPG  plans  to 
create  implementation  teams  focused  on  acquisition  and  human  resources  issues. 
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4.5  ALC  Number  3 


In  May  of  1992,  ALC  Number  3  initiated  their  first  software  process  assessment  in 
compliance  with  the  Embedded  Computer  Software  Program  Management  Plan  (ECS 
PMP).  This  assessment  is  the  most  recent  stage  in  their  ongoing  program  to  improve  their 
software  development  process. 

The  assessment  report  listed  the  assessment’s  objectives  as  identifying  key  areas  for 
process  improvement  and  proposing  a  framework  for  subsequent  improvement  actions.  In 
addition,  the  report  emphasized  concern  that  the  findings  were  not  intended  to  be  critical 
of  personnel  or  projects.  Finally,  although  the  results  of  the  assessment  were  considered 
private,  the  ALC  commander  agreed  to  release  both  the  maturity  level  and  findings. 

The  assessment  team  consisted  of  local  ALC  SEPG  personnel,  representatives  from 
other  ALCs,  and  an  observer  and  coach  from  the  SEI.  The  team  reviewed  five  software 
projects  during  the  assessment.  In  addition  to  in-depth  study  of  projects,  the  team  met 
with  45  functional  area  representative  from  all  areas  of  the  software  engineering  division. 

Sources  for  information  for  this  case  study  include  questionnaires,  the  assessment 
report,  and  telephone  conversations  with  ALC  personnel.  Questionnaires  were  sent  to 
software  engineering  process  group  (SEPG)  personnel  as  well  as  key  software  personnel 
from  several  representative  projects  within  the  software  engmeeiing  division  of  ALC 
Number  3.  A  summary  of  the  questionnaire  responses  for  ALC  Number  3  is  included  as 
Appendix  D.  The  next  section,  summary  of  assessment  findings,  reviews  the  process 
assessment  report.  After  the  assessment  report  discussion,  the  process  improvement 
program  section  outlines  the  ALC's  subsequent  process  improvement  efforts.  Telephone 
conversations  with  SEPG  leaders  and  members  were  the  sources  for  process  improvement 
information. 
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4.5.1  Summary  of  Assessment  Findings.  At  the  conclusion  of  the 
assessment,  the  team  graded  the  software  engineering  division  of  this  ALC  as  "emerging 
into  Maturity  Level  H  the  Repeatable  LeveL"  This  terminology  was  used  to  differentiate 
between  organizations  at  different  stages  within  the  first  maturity  level.  Essentially,  the 
assessment  team  found  relatively  few  unsatisfied  key  process  areas  for  Level  n.  In  fact, 
some  evidence  of  strong  Level  II  processes  and  even  Level  EDI  activities  were  discovered. 
Nine  areas  of  concern  were  noted.  The  first  three  finding  areas  were  the  only  unsatisfied 
Key  Process  Areas  for  a  Level  II  rating.  Within  these  shortfalls,  software  quality 
assurance  was  regarded  as  the  biggest  hurdle  between  ALC  Number  3  and  a  Level  II 
rating.  Training,  peer  reviews,  and  organizational  process  definition  represent  challenges 
for  the  third  or  Defined  level.  Finally,  human  resources,  test  equipment,  and  project 
resources  were  specific  areas  of  need  identified  by  the  assessment  team  rather  than  CMM 
key  process  areas.  The  following  list  includes  all  finding  areas  for  ALC  Number  3’s 
software  processes,  a  brief  description  of  each  domain,  and  ALC  Number  3's  related 
problem(s). 

1 .  Software  Quality  Assurance  (Software  quality  assurance  involves 
reviewing  and  verifying  the  software  products  and  activities  to  ensure  that  they 
comply  with  the  applicable  processes,  standards  and  procedures) 

Specific  Problem:  The  software  quality  assurance  functions  are  not  well- 
defined. 

2.  Software  Project  Planning  (Software  project  planning  involves  developing 
estimates  for  the  work  to  be  performed,  establishing  the  necessary 
commitments,  and  defining  the  plan  to  perform  the  work) 

Specific  Problem:  No  effective  mechanism  exists  to  ensure  that  software 
size,  cost,  and  schedule  estimates  are  consistent,  accurate,  and  used. 

3.  Software  Management  (Software  management  involves  tracking  and 
reviewing  the  software  accomplishments  and  results  against  documented 
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estimates,  commitments,  and  plans,  and  adjusting  these  based  on  the  actual 
accomplishments  and  results) 

Specific  Problems:  Critical  software  metrics  are  not  adequately  defined, 
recorded,  and  tracked  There  is  a  lack  of  a  mechanism  to  ensure  periodic 
management  review  of  each  project. 

4.  Training  (Training  involves  identifying  the  training  needs  of  the  organization, 
die  projects,  and  the  individuals,  and  developing  and  procuring  courses  to 
address  these  requirements) 

Specific  Problems:  There  is  a  lack  of  an  adequate  formal  training  program 
for  software  managers  and  practitioners.  The  formal  traiiiing  that  is  available  is 
often  not  coordinated  with  project  needs  or  schedules.  Some  mandated 
courses  arc  inappropriate  (not  relevant  to  the  software  division). 

5.  Human  Resources  (The  talented  people  that  make  up  an  organization  are  its 
greatest  resource.  The  way  it  rewards,  manages,  and  fosters  the  growth  of  its 
employees  is  crucial  to  its  lasting  success) 

Specific  Problems:  There  is  a  lack  of  a  technical  career  path  in  the  software 
division  for  engineers  and  technicians.  The  opportunities  to  broaden  one's 
career  path  by  moving  from  one  technical  position  to  another  are  limited.  High 
performance  is  not  adequately  rewarded.  Low  performance  is  noi  effectively 
managed. 

6.  Test  Equipment  (Test  equipment  includes  both  automatic  test  equipment, 
used  to  test  electronic  circuits,  and  the  wide  variety  of  computers  and 
equipment  used  to  test  the  operational  flight  programs) 

Specific  Problem:  Test  equipment  is  often  a  bottleneck,  and  is  sometimes 
inappropriate  for  testing  needs. 

7.  Project  Resources  (The  resources  required  to  successfully  complete  a 
software  project  need  to  be  acquired  in  a  timely  fashion.  For  most  software 
projects,  these  resources  include  manpower,  equipment,  and  tools.  Some 
projects  also  require  the  design  or  modification  of  hardware,  and  any  required 
parts  must  be  received  before  the  software  may  be  integrated) 

Specific  Problems:  The  process  for  acquiring  dime-store"  parts  is 
cumbersome  and  inefficient.  The  acquisition  of  software  engineering  tools  is 
often  impeded 
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8.  Peer  Reviews  (Peer  reviews  involves  a  methodical  examination  of  work 
products  by  the  products  by  the  producer’s  peers  to  identify  defects  and  areas 
where  changes  and  improvements  are  needed) 

Specific  Problem:  Peer  reviews  are  not .  xisisu.  onducted  and  the 
documentation  for  the  peer  review  process  is  lack*  ^ 

9.  Organizational  Process  Definition  (Organizational  process  definition 
involves  establishing  and  maintaining  a  standard  software  process  for  the 
organization,  for  use  by  the  projects  in  establishing  their  software  process.) 

Specific  Problems:  The  is  little  standardization  of  processes.  There  is  no 
standard  method  for  documenting  and  maintaining  processes.  A  central 
repository  for  software  processes  has  not  been  organized. 

4.52  Process  Improvement  Program.  ALC  Number  3's  process 
improvement  efforts  reflect  the  complexity  of  software  processes  and  the  difficulty  in 
starting  a  full-scale  organizational  effort  After  the  May  1992  assessment  SEPG  and 
assessment  team  members  spent  ten  months  in  carefully  publishing  their  report  and 
drafting  an  action  plan.  Since  the  assessment  report's  publication  in  March  of  1993,  SEPG 
leaders  have  described  the  process  group  as  "a  flurry  of  activity."  Working  under  the 
guidance  of  the  draft  action  report  15  Implementation  Teams  were  setup.  Each  team  is 
responsible  for  either  a  specific  Level  II  or  m  CMM  Key  Process  Area  (KPA)  or  a  non- 
CMM  finding  area.  Rather  than  concentrating  only  on  specific  finding  areas,  this 
aggressive  approach  ensures  that  all  Level  Q  and  III  KPAs  are  addressed. 

All  teams  function  with  the  same  basic  guidance.  Initially,  each  team  is  expected  to 
baseline  current  practices  and  organize  process  evaluation  techniques.  Once  underway, 
the  teams  implement  process  improvements  and  use  the  evaluation  techniques  for 
^feedback.  If  changes  are  beneficial,  they  will  be  baselined  and  the  process  will  be 
repeated. 

Each  implementation  team  meets  at  least  weekly.  One  major  issue  surfacing  in  most 
team  meetings  is  conformance.  Because  of  differing  requirements  in  the  many  types  of 
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software  projects  underway  in  Ai^C  Number  3,  teams  are  having  difficulty  reducing  the 
number  of  different  software  processes  and  tools  to  a  manageable  number.  One  example 
of  this  problem  is  the  different  software  development  tools  required  for  Automatic  Test 
Equipment  (ATE)  projects  and  Operational  Flight  Program  (OFP)  projects.  Each  type  of 
work  has  evolved  to  its  current  processes  and  tools  as  a  result  of  past  necessity,  and  while 
none  are  necessarily  incorrect,  the  number  of  different  methods  and  tools  must  be  reduced 
for  consistency. 

4JJ  Analysis  for  a  Relationship  between  Project  Success  and  SEI 
CMM  Ratings.  Our  first  objective  was  to  determine  if  a  relationship  exists  between  SEI 
Capability  Maturity  Model  ratings  and  project  success.  Section  three  of  the  questionnaire 
and  subsequent  follow-up  telephone  conversations  was  the  primary  source  of  information 
regarding  this  objective. 

All  five  respondents,  representing  the  four  projects  studied,  indicated  good  customer 
satisfaction.  Two  respondents,  representing  project  A,  stated  that  costs  were  within  or 
below  budget  estimates.  Two  other  respondents,  from  projects  C  and  D,  indicated  that 
costs  were  either  on-target  or  average.  Finally,  the  respondent  for  project  B  stated  that 
costs  had  not  changed  as  a  result  of  process  improvements.  The  respondent  believed  the 
project's  relatively  late  stage  of  development  to  be  the  reason  for  the  lack  of  change. 

In  terms  of  schedule,  respondents  from  project  A  indicated  the  effort  was  within 
schedule.  Project  B's  response,  once  again,  indicated  that  the  project  was  too  far  along  to 
be  affected  by  recent  process  improvements.  Project  C's  respondent  indicated  that  the 
project  was  behind  schedule  due  to  subcontractor  difficulties.  Finally,  project  D's 
respondent  did  not  indicate  anything  concerning  project  schedule. 

Only  the  respondents  from  projects  B  and  D  stated  that  the  process  improvements 
had  already  increased  project  success.  Both  respondents  indicated  that  more  customer 
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interaction  had  increased  customer  satisfaction.  Other  respondents  stated  that 
improvements  had  yet  to  affect  their  projects. 

As  far  as  their  expectations,  representatives  from  project  A  indicated  that  the 
process  improvements  would  not  likely  effect  their  project  As  reasons  for  not  expecting 
benefits  from  the  process  improvement  program,  project  A's  respondents  indicated  that 
they  had  already  addressed  some  of  the  improvements  and  the  program  was  nearing 
completion.  The  respondents  from  die  remaining  projects  each  felt  the  improvements  will 
affect  project  success  in  the  future.  Reasons  for  projecting  future  improvements  included 
improved  ability  to  handle  the  unexpected,  better  tools,  tracking,  and  teamwork. 

While  the  outlook  is  optimistic,  the  results  from  ALC  Nr~  ber  3‘s  projects  did  not 
indicate  a  conclusive  relationship  between  SEI  ratings  and  project  success.  In  addition  to 
determining  the  effect  of  process  improvements  on  project  success.  Chapter  Five 
compares  project  success  across  cases  in  search  of  more  evidence  of  a  relationship. 

4.5.4  Analysis  of  Organizational  Acceptance  and  Impact.  The  second 
objective  is  to  evaluate  the  organizational  acceptance  and  impact  of  the  SEI  assessment 
process.  Sections  one  and  three  of  the  questionnaire  and  the  assessment  report  are  the 
primary  sources  of  information  for  this  objective. 

Reasons  for  initiation  of  the  SEI  assessment/improvement  program  differed  among 
the  SEPG  respondents.  One  respondent  indicated  the  directorate  had  made  a  commitment 
to  quality  improvement.  Two  stated  that  the  Air  Force  mandate  was  the  only  reason  for 
the  assessment.  Clearly  the  concept  of  process  improvement  was  not  originated  locally 
within  this  ALC. 

SEPG  members  were  obviously  supporters  of  the  SEI  CMM  methodology.  Two 
respondents  identified  process  consistency  as  the  main  benefits  of  the  SEI’s  methodology. 
Another  response  stated  that  the  quality  improvement  concept  provided  a  voice  for 


workers  in  the  process.  Finally,  the  one  respondent  indicated  the  SEI  assessment 
methodology  had  the  advantage  of  thorough  assessments  and  provided  a  sound  process 
far  managing  software  projects. 

Each  respondent  also  recognized  several  disadvantages  of  the  CMM  Among  those 
disadvantages  were  the  CMM's  failure  to  consider  hardware  (system)  issues,  sometimes 
vague  criteria,  and  difficulty  understanding  the  CMM.  Despite  these  disadvantages,  all 
SEPG  members  agree  that  die  maturity  level  was  a  valid  representation  of  their 
organization's  ability  to  produce  software. 

Three  of  the  five  SEPG  respondents  indicated  that  worker  level  response  to  the  SEI 
process  assessmentAmprovement  program  was  positive.  Two  reasons  were  given  for  the 
positive  response.  First,  according  to  SEPG  responses,  workers  now  feel  that 
management  is  more  likely  to  listen  to  their  problems  and  ideas.  Second,  respondents 
stated  that  because  the  implementation  teams  mostly  came  from  worker  level  personnel, 
there  will  be  better  acceptance  of  their  changes.  The  two  disagreeing  SEPG  respondents 
felt  that  some  workers  have  taken  a  "wait  and  see”  or  "it  will  go  away"  position. 

Project  responses  to  the  issue  of  worker  level  reaction  were  very  different  Four  of 
the  five  responses  indicated  a  lack  of  worker  level  acceptance  of  the  process 
assessment/improvement  program.  These  four  respondents  stated  that  workers  were  not 
well  informed,  not  sure  of  what  would  happen  after  the  assessment,  not  involved,  or 
taking  a  "wait  and  sec"  approach.  Only  one  project  level  respondent  indicated  that  most 
workers  felt  like  pan  of  the  team.  None  of  the  respondents  indicated  that  all  of  the 
workers  were  negative,  but  clearly  there  are  groups  of  practitioners  that  were  not  behind 
the  initiative. 

Management's  response  to  the  assessment  and  improvement  program  also  drew 
mixed  opinions.  Three  of  five  SEPG  members  agreed  that  management  was  actively 
participating  in  the  program.  One  respondent  felt  that  some  branch  or  section  level 
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managers  seemed  to  think  their  portions  of  the  organization  were  doing  things  well  and 
didn't  pay  much  attention  to  the  efforts.  The  final  SEPG  respondent  indicated  that 
management  simply  continued  "business  os  usual."  The  project  respondents  also  had 
mixed  responses.  Two  of  five  project  respondents  stated  that  management  had,  although 
slowly,  accepted  the  assessment  results  and  actively  pursued  process  improvement.  Two 
remaining  project  respondents  indicated  that  management  was  simply  complying  with  the 
mandated  program  and  one  felt  that  management's  commitment  had  dwindled  since  the 
assessment 

To  summarize  the  second  objective,  SEPG  and  project  level  respondents  are  solidly 
behind  the  SEI  assessment/improvement  methodology.  SEPG  respondents  indicated  a 
much  more  positive  worker  level  response  and  acceptance  of  the  assessment  and 
improvements  than  indicated  by  project  level  respondents.  Roughly  half  of  both  sets  of 
respondents  indicated  that  management  was  solidly  behind  the  improvement  process.  The 
differences  between  perceptions  are  probably  a  result  of  respondents'  experience  working 
in  different  portions  of  the  organization  with  different  management  personnel  and  workers. 
In  addition,  the  perceived  lack  of  worker  level  support,  identified  by  project  level 
respondents,  indicates  a  lack  of  communication  throughout  the  organization  concerning 
the  improvement  effort 

4.5  J  Analysis  of  Organizational  Process  Improvement  Responses.  The 
third  research  objective  was  to  review  organizational  process  improvement  responses 
following  assessments.  Section  two  of  the  questionnaire  and  the  telephone  conversations 
with  SEPG  members  are  the  sources  of  information  for  this  discussion. 

The  creation  of  eighteen  Key  Process  Area  (KPA)  implementation  teams  is  the 
biggest  action  taken  as  a  result  of  ALC  Number  3's  assessment.  These  teams  are  not, 
however,  designed  to  simply  answer  the  assessment.  Instead,  a  team  has  been  created  for 


each  KPA  associated  with  Levels  II  and  HI  of  the  CMM.  The  charter  of  each  team  is  to 
baseline  their  specific  portions  of  the  software  process,  develop  a  means  to  evaluate  their 
process,  and  implement  and  test  improvements  to  the  process  in  order  to  meet  the  related 
goals  of  the  CMM.  One  of  the  challenges  faced  by  these  teams  was  the  issue  of 
conformance.  The  widely  different  types  of  projects  within  the  ALC  have  made  it  difficult 
to  reduce  the  number  of  different  processes  and  tods  in  use. 

At  the  project  level,  four  of  five  respondents  specifically  indicated  that  a  newly 
formalized  peer  review  program  has  been  put  in  place.  One  of  these  four  also  indicated 
that  weekly  status  meeting  had  improved  project  tracking.  These  two  improvements  are 
directed  toward  improving  the  software  management  and  peer  review  findings  areas. 
SEPG  personnel  identified  the  creation  of  the  process  area  teams,  but  this  action  has  not 
yet  affected  most  project  level  personnel. 

The  goal  of  ALC  Number  3  is  to  achieve  CMM  Level  HI  by  their  next  assessment  in 
September  of  1994.  Several  improvements  have  already  been  put  in  place  on  the  way  to 
that  goal.  Most  importantly,  the  SEPG  and  implementation  teams  have  used  the  CMM  to 
develop  a  framework  for  process  understanding  and  improvement.  Project  level 
improvements  also  include  formal  peer  reviews,  regular  status  meetings,  identification  and 
documentation  of  processes,  and  the  identification  of  bottlenecks  in  processes.  ALC 
Number  3  will  likely  achieve  their  goal  through  the  customized  improvement  program 
established  by  the  SEPG.  More  important,  a  cadre  of  supporters  has  developed  to 
improve  the  way  ALC  Number  3  develops  and  supports  suL ..  ore.  Finally,  ALC  Number 
3  still  faces  the  challenge  of  gaining  and  holding  management  commitment  and  worker- 
level  buy-in  of  the  process  improvement  program. 


V.  Cross-Case  Analysis 


5.1  Relationship  Between  SEI  Ratings  and  Project  Success 

The  first  objective  of  our  research  was  to  determine  if  SEI  ratings  are  related  to 
project  success.  In  the  case  of  ALC  Number  1,  the  recency  of  their  latest  assessment 
(March  1993)  made  it  difficult  to  conclude  that  a  relationship  existed  between  their  SEI 
maturity  rating  and  project  success.  In  addition,  the  resulting  improvement  efforts  have 
not  been  implemented  yet  However,  several  on-going  efforts,  somewhat  re-focused  as  a 
result  of  the  most  recent  assessment  were  identified  by  some  of  the  questionnaire 
respondents  as  having  the  potential  to  affect  the  success  of  the  projects  we  looked  at. 

The  analysis  of  ALC  Number  2  indicated  that  a  relationship  did  exist  between 
project  success  and  the  improvements  implemented  as  a  result  of  an  assessment  Process 
improvements  regarding  requirements  definition,  project  planning,  and  project 
management  have  increased  customer  interaction,  improved  cost  and  schedule  estimates, 
and  helped  to  solve  problems  while  they  were  still  minor.  Although  it  we  will  not  learn  if 
the  organization  has  increased  to  a  higher  maturity  level  until  the  next  assessment  the 
improvement  are  surely  a  step  toward  achieving  their  current  goal  -  CMM  Level  II. 

These  recent  improvements  have  already  led  to  increased  customer  satisfaction,  product 
quality,  and  thus  project  success. 

ALC  Number  3's  results  do  not  conclusively  indicate  a  relationship  between  project 
success  and  maturity  level.  The  differences  in  the  responses  from  the  four  projects 
suggest  that  effects  of  the  assessment  recommendations  can  vary  within  a  single 
organization,  making  it  difficult  to  conclude  that  a  relationship  exists  between  project 
success  and  maturity  level.  Two  of  four  projects  studied  felt  that  the  process 
improvements  had  led  to  increased  customer  satisfaction,  but  the  remaining  projects 
indicated  no  effect.  Respondents  were,  however,  optimistic  that  improvements  were  likely 
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to  increase  the  future  success  of  these  projects.  The  only  exception  to  this  optimism  was 
the  in  the  case  of  one  respondent  whose  project  is  expected  to  end  piior  to  implementation 
of  many  of  the  process  improvements. 

The  analyses  of  ALC  Number  1  and  ALC  Number  3  were  showed  no  conclusive 
relationship  between  project  success  and  maturity  levels.  The  reason  for  lack  of 
conclusive  evidence  in  ALC  Number  1  was  largely  due  to  short  tinw  between  the 
assessment  (March  1993)  and  this  research  effort.  Although  some  ALC  Number  3 
projects  suggested  that  process  improvements  had  led  to  increased  customer  satisfaction, 
others  indicated  no  relationship.  Differences  between  the  responses  can  not  be  attributed 
to  process  improvement  efforts,  worker  level  response,  or  management  acceptance  of  the 
improvement  program.  Therefore,  a  relationship  between  project  success  and  maturity 
level  or  process  improvements  cannot  be  determined. 

5.2  Organizational  Acceptance  and  Impact  of  the  SEI  Assessment  Process 

Overall,  ALC  Number  1  reacted  favorably  to  being  assessed,  including  the  findings 
in  the  on-going  process  improvement  program,  and  generally  maintained  an  enthusiastic 
attitude  toward  the  SEI  assessment  process.  Management  has  consistently  supported 
assessment  efforts  and  remains  committed  to  the  CMM  and  its  guiding  influence  on  the 
overall  organizational  process  improvement  plan. 

The  reaction  of  ALC  Number  2  to  the  assessment  and  improvement  process  has 
varied  considerably  through  the  organization.  All  respondents  indicated  that  resultant 
maturity  level  is  a  valid  representation  of  their  organization's  ability  to  produce  and 
support  software.  Some,  however,  stated  that  the  main  issue  is  not  process  improvement, 
but  product  quality  and  customer  satisfaction.  In  the  view  of  these  respondents,  process 
improvement  is  valuable  only  to  the  extent  that  it  improves  product  quality  and  customer 
satisfaction.  Worker  level  response  varied  from  cautious  to  active  support.  Management 
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support  was  similarly  categorized  None  of  the  respondents  indicated  that  a  majority  of 
personnel  (worker  level  or  management)  were  not  supportive,  merely  some  of  the 
personnel.  Respondents  also  suggested  that  worker  support  had  increased  with  rime.  All 
respondents  were  optimistic  about  the  benefits  of  the  program,  and  project  level 
respondents  indicated  quality  and  customer  satisfaction  benefits. 

ALC  Number  3's  reaction  to  the  assessment  and  improvement  process  has  also 
varied  considerably  through  die  organization.  All  respondents  were  clearly  supporters  of 
the  SEI  methodology  and  indicated  that  maturity  is  a  valid  representation  of  their 
organization's  ability  to  produce  and  support  software.  SEPG  respondents  viewed 
management  and  worker  level  reaction  to  the  assessment  as  very  positive.  Project 
respondents  indicated,  however,  that  some  members  of  both  the  management  and  the 
worker  levels  were  skeptical  about  the  initiative. 

Respondents  from  all  three  ALCs  stated  that  the  maturity  rating  was  considered  a 
valid  indication  of  their  organization's  ability  to  produce  and  support  software.  While 
minor  skepticism  concerning  the  value  of  the  assessment  process  was  noted  at  ALC 
Number  1,  the  other  two  ALCs  identified  significant  doubt  among  some  personnel.  At 
both  ALC  Number  1  and  ALC  Number  2,  worker  level  support  for  the  program  was 
perceived  as  less  solid  than  management  support  Reasons  for  this  lack  of  support  range 
from  caution  on  the  workers  part  to  belief  that  the  process  improvements  are  "just  the 
latest  fad."  All  three  ALCs  met  the  spirit  of  the  SEI  methodology  by  maintaining  the  non¬ 
attributive  atmosphere  throughout  the  assessment  process. 

ALC  Number  1  accepted  the  assessment  the  most  favorably  of  the  three  ALCs.  This 
is  probably  due  to  several  factors.  First,  ALC  Number  1  had  been  assessed  once  before 
and  was  familiar  with  the  assessment  process.  Second,  ALC  Number  1  has  been 
interested  in  software  process  improvement  for  several  years  and  had  already  defined  an 
action  plan  and  set  up  the  mechanism  for  implementing  and  measuring  process 
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improvements.  Third,  the  initial  assessment  was  performed  at  the  request  of  ALC  Number 
1  and  was  not  the  esulr  of  a  Command  mandate.  The  only  assessments  performed  at 
ALCs  2  and  3  were  the  direct  result  of  AFMC's  mandate  that  all  ALC  be  assessed. 

53  Organizational  Process  Improvement 

As  mentioned  above,  ALC  Number  1  has  been  working  toward  an  on  going  process 
improvement  program  since  1989.  Most  of  the  improvements,  both  recent  and  planned, 
are  the  result  of  efforts  not  related  to  the  most  recent  (March  1993)  assessment. 

However,  several  process  improvements  have  been  undertaken  as  a  direct  result  of  the 
March  1993  assessment.  A  process  improvement  plan  is  maintained  by  the  SEPG  that 
was  formulated  using  the  results  of  the  1989  assessment  and  internal  recommendations  for 
improvements.  The  rapid  inclusion  of  the  March  1993  assessment  findings  in  the  July 
1993  version  of  the  process  imp;  -  ement  action  plan  illustrates  die  commitment  to 
improvement  efforts  and  specifically,  high  regard  for  the  SEI  CMM  As  stated  already, 
the  focus  is  on  continuous  process  improvement,  not  response  to  specific  findings.  The 
SEI  CMM  is  merely  used  to  measure  the  overall  cumuladve  success  of  ALC  Number  I  s 
improvement  efforts. 

Tht  process  improvement  efforts  at  ALC  Number  2  were  centered  around  the 
findings  from  their  1992  SEI-assisted  process  assessment.  The  first  three  finding  areas: 
requirements  definition,  project  planning,  and  poject  planning  and  oversight  were  the  ftr.i 
to  receive  attention.  Process  action  teams  were  assembled  to  address  each  Key  Process 
Area  (KPA).  Increased  customer  interaction  regarding  software  requirements  and  more 
thorough  tracking  of  cost  and  schedule  information  directly  resulted  from  these  teams 
These  initial  improvements  have  already  increased  customer  satisfaction  and  thus  project 
success. 
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Although  these  accomplishments  are  impressive,  they  were  just  the  first  step  in  ALC 
Number  2's  process  improvement  program.  Teams  were  recently  teams  organized  to 
address  the  two  remaining  Level  II  KPAs.  In  addition  to  these  CMM-related 
improvements,  the  SEPG  plan:  to  establish  working  groups  for  human  resources  and 
acquisition  issues. 

While  the  pace  of  improvement  was  not  as  rapid  as  some  personnel  hoped  for,  the 
early  success  appeared  to  have  helped  build  organizational  support.  The  process 
improvement  framework  in  place  at  ALC  Number  2  vill  likely  lead  to  attainment  of  their 
modified  goal  -•  Level  II  maturity  by  September  1994. 

ALC  Number  3  approached  software  process  improvement  much  as  ALC  Number  1 
did.  Rather  than  using  the  assessment  results  as  their  only  guide,  they  established  teams  to 
address  each  KPA  fen:  Levels  H  and  m  of  the  CMM  as  well  as  other  findings  areas  not 
specifically  pan  of  the  CMM.  This  broad  effort  indicates  a  major  commitment  by  both 
management  and  personnel  to  continue  process  improvement.  While  opinions  about 
tangible  results  are  mixed,  most  agree  that  increased  customer  interaction  and  peer 
re/iews  are  positive  steps.  Although  ALC  Number  3  s  effort  was  the  last  to  begin,  they 
have  clearly  embraced  the  SEI  assessment/improvement  program  and  will  likely  achieve 
their  goal  of  Level  HI  by  September  of  1994. 

Each  organization  implemented  its  process  improvements  in  a  slightly  different 
manner.  All  three  organizations,  however,  agreed  that  the  assessment  results  and  the 
CMM  should  not  be  used  as  the  only  basis  for  process  improvement.  Top  management  of 
each  ALC  have  given  complete  support  to  each  effort.  While  some  indication  of 
skepticism  or  lack  of  support  was  found  among  management  and  workers  at  each  ALC, 
stronger  support  from  each  group  was  evident  at  ALC  Number  1 .  Their  support  may  be 
the  result  of  the  length  of  time  ALC  Number  l's  program  has  been  underway.  Process 
improvement  efforts  at  both  ALC  Number  2  and  ALC  Number  3  have  strong  support,  but 
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also  more  skeptics,  especially  among  workers.  Most  respondents  at  each  location  felt  that 
suppon  continued  to  grow  with  the  program. 

Each  ALC  organized  their  personnel  into  implementation  teams  for  each  CMM  key 
process  area  and/or  other  areas  designated  for  improvement.  Use  of  project  level 
personnel  on  these  teams  was  noted  to  benefit  worker  level  "buy-in"  or  acceptance  of  the 
process  improvement  effort 

While  each  effort  is  different,  each  ALC  has  customized  their  process  improvement 
efforts  to  their  organizational  needs.  None  of  the  ALCs  is  following  the  CMM  like  a 
cookbook.  Each  SEPG  has  studied  the  CMM  as  part  of  a  comprehensive  software 
process  improvement  effort  ALC  Number  1  is  surely  further  along  in  the  process  than  the 
other  two,  but  based  on  improvements  already  underway  and  planned,  each  will  likely 
achieve  their  immediate  process  maturity  goals. 


VI.  Conclusions 
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6.1  Summary 

The  primary  objective  of  this  study  was  to  determine  if  a  relationship  existed 
between  maturity  level  and  project  success.  Project  success  was  defined  as  the 
combination  of  quality,  cost,  and  schedule.  In  addition,  we  examined  the  impact  on  the 
assessed  organizations  of  the  SEI CMM  assessment  and  follow-on  process  improvement. 
Finally,  we  reviewed  organizational  process  improvements  following  the  assessment(s). 
The  research  was  accomplished  through  a  combination  of  information  gathering 
techniques  and  data  analysis.  A  literature  review,  both  within  and  external  to  the 
Department  of  Defense,  summarized  current  software  process  research,  discussed  the 
CMM  in  detail,  presented  case  studies  using  the  SEI  CMM,  and  introduced  software 
project  success  criteria.  Using  information  gathered  from  three  Air  Force  software 
organizations,  we  addressed  each  research  objective. 

6.2  Results 

For  the  most  part,  it  cannot  be  concluded  that  SEI  maturity  level  is  an  indication  of 
project  success  for  the  organizations  studied.  This  does  not  mean  that,  in  general, 
maturity  level  is  not  a  good  indicator  of  project  success,  but  rather  for  the  organizations 
studied,  it  could  not  be  conclusively  determined  that  CMM  ratings  were  an  indicator  of 
project  success. 

The  participating  projects,  within  die  organizations  studied,  could  not  determine 
^whether  their  success  was  the  result  of  improvements  implemented  as  a  result  of  their 
assessments,  or  the  result  of  other  factors  such  as  personnel  capability  or  previous  domain 
experience.  For  some  of  the  respondents  from  the  individual  projects,  the  process 
improvements  resulting  from  the  most  recent  assessment  should,  when  implemented, 
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influence  their  projects'  success.  For  others,  the  projects  were  either  recently  begun, 
nearly  complete,  or  the  process  improvements  were  not  yet  fully  implemented. 

All  of  die  organizations  indicated  that  their  organizations  accepted  and  supported 
the  assessment  process  and  findings.  They  believed  that  the  assessments  accurately 
measured  their  organizations'  software  process  maturity  level.  Also,  having  an  outside 
group  of  software  experts,  from  the  SE1,  assess  the  organization  was  thought  to  add 
credibility  to  the  assessment  results.  In  addition,  having  in-house  personnel  participate  in 
the  assessment  increased  the  organizational  acceptance  of  the  findings  and 
recommendations. 

Organizations  that  have  focused  on  continuing  process  improvement,  seemed  to  be 
more  open  to  the  assessment  process,  recommendations,  and  follow-up  action  plan. 

Those  organizations  that  were  relatively  new  to  the  software  process  improvement,  were 
less  enthusiastic.  Respondents  indicated  that  management  and  worker  level  support  was 
stronger  in  organizations  that  have  been  actively  involved  in  an  ongoing  process 
improvement  program.  Finally,  all  of  the  organizations  indicated  that  there  were 
individuals  who  were  skeptical  of  the  assessment  process  initially  but  that  this  skepticism 
eventually  faded. 

Finally,  our  third  objective  was  to  review  the  organizational  process  improvement 
efforts  following  the  assessments.  Each  organization  has  a  process  improvement  plan  in 
place,  but  there  were  differences  among  them.  ALC  Number  l's  process  improvement 
plan  does  not  direcdy  focus  on  the  SEI  assessment  findings,  but  instead  seeks  to 
implement  improvements  based  on  their  internal  requirements.  ALC  Number  2’s  plan  is 
built  upon  the  SEI  assessment  findings  report  which  contains  recommendations  outside 
the  rc.’m  of  the  CMM.  ALC  Number  3  also  bases  its  process  improvement  plan  on  the 
SEI  CMM  and  has  established  a  working  group  for  each  Level  II  and  III  key  process  area 
(KPA). 
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Although  this  study  could  not  conclusively  establish  a  relationship  between  project 
success  and  SEI  maturity  levels,  we  have  shown  that  an  organization  that  has  invested 
effort  in  developing  a  process  improvement  infrastructure  can  use  the  CMM  as  a  guide  to 
develop  an  on-going  process  improvement  program.  The  basic  tenet  of  process 
improvement  is  that  the  product  can  only  be  as  good  as  the  process,  and  in  each  case, 
respondents  from  each  case  studied  were  confident  that  process  improvements  will 
concurrently  lead  to  higher  maturity  levels  and  increased  project  success. 

During  the  research  design  portion  of  our  effort,  we  hoped  to  find  quantitative 
information  demonstrating  or  disproving  the  relationship  between  project  success  and  SEI 
CMM  ratings.  Because  this  information  was  not  yet  available,  we  chose  to  evaluate  the 
relationship  in  a  more  qualitative  manner  using  a  few  Air  Force  organizations  that  have 
been  assessed.  This  study  was  an  important  first  step  in  evaluating  the  value  of  the  SEI's 
CMM  to  the  Air  Force.  As  process  improvement  programs  and  related  software  projects 
mature,  more  conclusive  information  regarding  this  relationship  will  become  available. 

6  .4  Recommendations  for  Further  Research 

Data  concerning  our  primary  thesis  objective,  to  determine  the  relationship,  if  any, 
between  SEI  CMM  ratings  and  project  success,  was  inconclusive  primarily  because  of  the 
relative  immaturity  of  the  process  improvement  efforts  at  some  ALCs.  As  a  result  of  the 
newness  of  the  efforts,  many  of  the  projects  that  participated  in  the  study  had  either  not 
yet  been  affected  by  process  improvements,  or  they  were  too  far  along  to  be  affected  by 
changes.  Further  research  needs  to  develop  more  comprehensive  data  on  project  success 
and  process  improvement  efforts,  preferably  from  organizations  that  have  been  through 
several  iterations  with  the  CMM.  In  addition,  we  recommend  that  the  organizations  used 
in  this  research  be  studied  because  the  effort  required  to  collect  the  preliminary  assessment 
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and  process  improvement  data  is  too  time  consuming  to  allow  for  a  signi  :ant  research 
effort  beyond  what  was  accomplished  in  the  thesis. 


6.4.1  Study  Replication.  Using  the  same  methodology,  replicate  the  research 
with  the  same  organizations  using  the  same  projects.  This  strategy  will  incorporate  the 
added  maturity  of  the  process  improvement  programs  and  its  effect  on  projects.  Using  the 
same  projects  and  organizations  will  be  the  basis  for  interesting  comparison  related  to  all 
three  research  objectives.  Using  the  same  organizations  will  demonstrate  the  evolution  of 
incorporation  and  acceptance  of  the  total  quality  management  within  the  organizations. 

6.4.2  Study  Replication  with  Different  Projects.  Using  the  same 
methodology,  replicate  the  research  with  the  same  organizations  using  the  several  different 
individual  projects.  This  strategy  will  also  allow  for  added  maturity  of  the  process 
improvement  programs  and  its  effect  on  projects.  Selecting  projects  that  arc  more  likely 
to  be  affected  by  the  process  assessment/improvement  will  provide  better  information 
concerning  the  suggested  relationship  between  project  success  and  SEI CMM  ratings. 
Once  again,  using  the  same  organizations  will  demonstrate  the  evolution  of  incorporation 
and  acceptance  of  the  total  quality  management  within  the  organizations. 

6.43  In-Depth  Single  Case  Study.  An  in-depth  case  study  of  any  of  the 
organizations  included  in  this  study  would  yield  valuable  insight  into  the  internal  software 
process  improvement  plan  development  An  in-depth  study  of  ALC  Number  1  would  be 
especially  interesting  because  their  process  improvement  program  is  the  roost  defined. 
They  have  had  much  success  in  implementing  improvements  and  have  advanced  to  a  Level 
II  as  a  result  A  study  of  this  scope  would  enable  the  researchers  to  interview  project  and 
SEPG  personnel  extensively,  thereby  gathering  more  detailed  data  on  project  success, 
process  improvement  progress,  and  organizational  acceptance  of  the  assessment.  As  an 
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initial  effort  in  this  area,  the  scope  of  the  current  study  was  limited  to  &  higher  level  survey 
of  die  three  organizations  that  touched  on  issues,  and  not  specific  findings. 


Appendix  A:  Research  Questionnaire 


General  Information: 

Date: _ 

Interviewee  Name:  _ 

Interviewee  Organization/Position : 


Interviewee  Phone  Number.  _ 

Project  Information 

Project  Name:  _ 

Brief  Statement  of  Project  Purpose: 


Cost:  _ 

Number  of  Personnel  Involved: 
Scheduled  Length  of  Project  _ 
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Section  1 .  Motivation  for  Process  Evaluation! Improvement  ( ALC  Level ) 

a.  What  factors  led  to  the  initiation  of  a  process  improvement  program? 


b.  What  factors  led  specifically  to  SEI  involvement? 


c.  What  is  your  overall  opinion  of  the  SEI  assessment  process/methodology? 

i 

(l)  Advantages: 


(?)  Disadvantages 


d.  Is  maturity  level  viewed  as  a  legitimate  representation  of  your  organization's  ability  to 
produce/support  software? _ 


e.  For  which  reasons  is  it  valuable  to  strive  for  higher-level  ratings?  (Circle  all  that  apply) 

(1)  Perception  (of  improvement) 

(2)  Improved  Product  (Quality,  Cost,  etc.) 

(3)  Headquarters  Mandate  (Required  to  stay  in  business) 

(4)  None/Other _ 

Explain  your  answer. _ 
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Section  2.  General  Process  Improvement! Assessment  Issues  (AJJC  and 
Project  Level) 

a.  Was  the  evaluation  accomplished  entirely  by  the  SEI,  through  an  SEI-assisted  team,  or 
a  self-assessment? 


b.  What  process  improvements/actions  have  been  taken  solely  as  a  result  of  the 
assessment  process? 


c.  What  other  process  improvements/changes  have  been  undertaken  within  the  same  time 
frame? 


d.  What  benefits  have  been  realized  through  the  SEI  Assessment  Process? 


e.  What  process  problems  have  been  identified? 
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Section  3.  Impact  on  Project  Success  (ALC  and  Project  Level) 


a.  At  what  level  was  the  organization  rated? 


I 


b.  What  actions  taken  as  a  result  of  the  SEi  assessment(s)  have  been  part  of  this  project? 


c.  How  successful  was  (is)  this  project?  (Please  address  the  following  areas) 

(1)  Customer  satisfaction 

(2)  Cost 

(3)  Schedule 

(4)  Other  areas? 


d.  How  have  the  post-SEI  assessment  actions  affected  the  success  of  this  project  in  the 
following  areas?  (Please  address  the  following  areas) 

(1)  Customer  satisfaction 

(2)  Cost 

(3)  Schedule 

(4)  Other  areas? 


e.  Do  you  feel  the  proposed  improvements  will  directly  result  in  benefits  to  this  project? 
Why? 


Section  4.  Human  Factors  ( ALC  and  Project  Level ) 

a.  What  was  the  worker  level  feeling  concerning  the  SEI  assessment? 


c.  How  thoroughly  has  management  embraced  the  SEI  assessment  results/suggestions 
(active  or  just  filling  a  square)? 


d.  What,  if  any,  negative  effects  have  been  realized  as  a  result  of  the  SEI  assessment? 


e.  Has  the  non-attributive  environment  been  maintained?  (or  has  head-hunting  taken 
place) 


f.  Has  the  status  of  these  factors  changed  since  your  process  improvement  efforts  were 
initiated? 


Appendix  B:  ALC  Number  1 

This  appendix  contains  a  complete  listing  of  the  questionnaire  responses  for  ALC 
Number  1. 


Section  1 .  Motivation  for  Process  Evaluation! Improvement  ( ALC  Level ) 
a.  What  factors  led  to  the  initiation  of  a  process  improvement  program? 


RESPONDENT 

RESPONSE 

SEP G  1 

Division  chief  attended  the  1989  SEI  Affiliates  Symposium.  Wc  had  been 
working  since  1987  and  assessments  looked  like  they  would  provide  focus. 

SEPG2 

Management  directed  as  a  result  of  a  quality  symposium  several  years  ago 

Project  1 

N/A 

Project  2 

N/A 

Project  3 

N/A 

b.  What  factors  led  specifically  to  SEI  involvement? 


RESPONDENT 

RESPONSE 

SEPG  1 

Same  as  above. 

SEPG2 

Management. 

Project  1 

N/A 

Project  2 

N/A 

Project  3 

N/A 

c.  What  is  your  overall  opinion  of  the  SEI  assessment  process/methodology? 
(1)  Advantages 


RESPONDENT 

RESPONSE 

SFPG  1 

The  new  method  is  a  major  improvement  We  were  the  Alpha  1  site  for  the 
new  process.  The  findings  are  now  CMM  based.  Also,  it  Jivolved  over  40% 
of  tne  organization  so  the  findings  are  seen  as  having  validity. 

SEPG  2 

Outside  perspective.  Knowledge  of  S/W  engineering  processes  procedures, 
and  concepts. 

Project  1 

N/A  | 

Project  2 

N/A 

Project  3 

N/A 

(2)  Disadvantages 


RESPONDENT 

RESPONSE  i 

SEPG  1 

The  assessment  itself  is  almost  two  weeks  long  and  very  intense.  At  times  it 
almost  felt  like  an  audit. 

SEPG  2 

Terminology.  Lack  of  detailed  understanding  of  our  specific  software  type 
etc. 

Project  1 

N/A 

Project  2 

N/A 

T.oject  3 

N/A 

d.  Is  maturity  level  viewed  as  a  legitimate  representation  of  your  organization’s  ability 
to  produee/support  software? 


RESPONDENT 

RESPONSE 

SEPG  1 

I  think  it  is.  We're  getting  more  organized  and  people  know  their  part.  As 
-we  grow  in  maturity  we  will  reduce  our  need  .or  heroes. 

SEPG  2 

For  the  moct  pan-yes. 

Project  1 

N/A 

Project  2 

N/A 

Project  3 

N/A 

e.  For  which  reasons  is  it  vJuable  to  strive  for  higher-level  ratings?  (Circle  all  that  apply) 

(0  Perception  (cf  improvement) 

(D  improved  Product  (Quality,  Cost,  etc.) 

(3)  Headquarters  Mandate  (Required  to  stay  in  business) 

(4)  None/Other 

Explain  your  answer 


RESPONDENT 

RESPONSE 

SEPG  1 

Improved  prod  act.  Headquarters  mandate:  We  started  this  before  any 
mandates  but  the  mandate."  have  helped  provide  attention  and  focus  oc 
process  improvement. 

StiPCi  2 

Improved  produce  The  SEI  "higher  level"  inherently  reil*cts  common-sense 
aspects  of  process  improvement  in  a  structu  ^d  framework(CMM). 

Project  1 

N/A 

Project  2 

N/A 

Project  3 

N/\ 

Section  2.  General  Process  Improvement! Assessment  Issues  (ALC  and 
Project  Level) 

a.  Was  the  evaluation  accomplished  entirely  by  the  SE1,  through  an  SEI-assisted  team,  or 
a  self-assessment? 


RESPONDENT 

r  RESPONSE 

SEPG  1 

Our  first  assessment  (Mar  90)  was  a  self  assessment  with  one  SE1  member. 

Our  second  assessment  (March  93)  was  the  Alpha  1  for  the  SETs  new  process. 

7  SEI  members,  2  SEI  observers,  2  ALC  members. 

SEPG2 

Entirely  by  SEI  personnel. 

Project  1 

SEI-Two  of  us  were  on  the  team. 

Project  2 

Entirely  by  the  SEI. 

Project  3 

SEI  only. 

b.  What  process  improvements/acdons  have  been  taken  solely  as  a  result  of  the 
assessment  process? 


RESPONDENT 

RESPONSE 

SEPG  1 

We  continually  monitor  our  fir<‘  -gs  and  what  we're  doing  to  correct  them. 
Basically  everything  relates  to  the  assessment. 

SEPG  2 

After  the  initial  self  assessment  the  division  management  steering  team  was 
organized  and  has  since  become  a  key  aspect  of  our  efforts. 

Project  1 

We  have  had  several  lunches  to  discuss  improvement  issues.  We  have  re¬ 
focused  some  on-going  issues  to  be  organization-wide. 

Project  2 

Speaking  for  my  section  only,  we  have  begun  having  peer  reviews  and  are  in 
the  process  of  developing  better  &  doormen  table  metrics  for  our  projects. 

Project  3 

Identify  training  needs  for  all  personnel.  Monthly  meetings  to  define  the 
organization's  standard  software  process. 

c.  Wha:  other  process  improvements/changts  have  been  undertaken  within  the  same  time 
frame? 


RESPONDENT 

RESPONSE 

SEPG  1 

All  improvements  go  througn  our  steering  team. 

SEPG  2 

Many.  The  organization  has  grown  tremendously  over  the  past  5  years. 

Inherent  in  this  growth  was  the  necessity  to  define  our  processes  and  improve 
the  way  in  which  we  function. 

Project  1 

All  improvement  issues  are  related. 

Project  2 

Just  piior(about  4-6  months)  to  the  assessment  wc  started  using  Earned  Value 
Reports  &  Chans'  to  track  our  projects.  This  should  help  to  establish  a  lot  of 
necessary  data  &  history. 

Project  3 

Development  of  database  to  track  software  media  and  documentation. 
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-7.  ' 


d  What  benefits  have  been  realized  through  the  SEI  Assessment  Process? 


RESPONDENT 

RESPONSE 

SEPG1 

Return  on  Investment  of  5.9  to  1 .  Helped  morale. 

SEPG2 

Outside  perspective  of  process  problems  or  deficiencies.  The  findings  (good 
&  bad)  can  be  related  directly  to  a  structured  framework,  i.e.  the  CMM. 

Project  1 

We  have  identified  areas  of  improvement  and  more  aware  of  our  current 
process. 

Project  2 

Mostly,  the  areas  where  we  were  weak  have  been  identified  an  other  areas 
where  we  were  lacking  process  were  brought  out.  It  was  also  encouraging  to 
see  what  we  were  doing  well. 

Project  3 

Everyone  realizes  where  our  weaknesses  are  and  strives  to  improve  in  those 
areas. 

e.  What  process  problems  have  been  identified? 


RESPONDENT 

RESPONSE 

SEPG  1 

Lack  of  tailoring  guidelines  or  documentation.  Undefined  ad-hoc  processes. 

SEPG2 

Problems  have  been  identified  and  documented  in  the  findings  report 

Project  1 

Very  difficult  to  collect  data  for  metrics. 

Project  2 

Lack  of  peer  reviews,  not  enough  clearly  defined  and  rigidly  documented 
process  plans. 

Project  3 

No  standard  organization  process  defined.  Training  needs.  Peer  reviews  not 
accomplished. 

Section  3.  Impact  on  Project  Success  (ALC  and  Project  Level) 
a.  At  what  level  was  the  organization  rated? 


RESPONDENT 

RESPONSE 

SEPG  1 

March  90--Level  1.  March  93-Level  2. 

SEPG  2 

Level  2. 

Project  1 

Level  2. 

Project  2 

Level  2  with  attributes  of  level  3. 

Project  3 

Level  2. 
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b.  What  actions  taken  as  a  result  of  the  SEI  assessment s)  have  been  part  of  this  project? 


RESPONDENT 

RESPONSE 

SEPG  1 

N/A 

SEPG2 

N/A 

CSSOHH 

We  have  re-focused  to  be  organizational. 

Project  2 

Nothing  specifically  because  our  projects  are  such  short  term.  However,  the 
overall  scope  of  our  projects  is  being  refined. 

None  yet 

c.  How  successful  was  (is)  this  project?  (Please  address  the  following  areas) 

(1)  Customer  satisfaction 

(2)  Cost 

(3)  Schedule 

(4)  Other  areas? 


RESPONDENT 

RESPONSE 

SEPG  1 

SEPG  2 

EZ3&30HHI 

Project  2 

Again,  our  projects  are  such  small  scale  that  it  is  too  early  to  measure  any 
substantive  effect 

Project  3 

Customers  are  quite  satisfied  with  software  mainienunc  activity.  Cost  is  a 
non-issue.  Software  maintenance  generally  accomplished  ahead  of  schedule. 

d.  How  have  the  post-SEI  assessment  actions  affected  the  success  of  this  project  in  the 
following  areas?  (Please  address  the  following  areas) 

(1)  Customer  satisfaction 

(2)  Cost 

(3)  Schedule 

(4)  Other  areas? 


RESPONDENT 

RESPONSE 

SEPG  1 

S'  PG  2 

I23E3S9HH 

j  Project  2 

None  have  been  drastically  affected  yet,  but  as  we  continue  to  develop  nev 
processes  &  refine  old  ones  everything  should  continue  to  improve. 

ESggi— 1 

No  impact  at  this  lime. 
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c.  I>>  you  feel  the  proposed  improvements  will  directly  result  in  benefits  to  this  project? 
Why? 


RESPONDENT 

RESPONSE 

SEPG1 

N/A 

SEPG2 

N/A 

Project  1 

N/A  | 

Project  2 

Yes,  because  as  we  continue  to  improve  our  successors  it  is  likely  that  the 
customer  will  set  more  for  the  money. 

Project  3 

Yes,  because  personnel  will  be  better  trained.  Software  development  will 
result  in  more  maintainable  code. 

Section  4.  Human  Factors  (ALC  and  Project  Level) 
a.  What  was  the  worker  level  feeling  concerning  the  SEI  assessment? 


RESPONDENT 

RESPONSE 

SEPG  1 

From  my  viewpoint,  having  been  through  the  last  assessment,  our  people  are 
prowl  of  what  they've  done  and  they’ve  embraced  process  improvement 

SEPG2 

As  far  as  I  know,  the  workers  have  agreed  with  and  accepted  the  assessment 
findinRs/rcsulis— {good  and  bad). 

Project  1 

We  realized  the  necessity  of  it  Exited  about  the  competition.  Anticipation 
what  will  come  out  of  it 

Project  2 

Pretty  good  for  the  most,  some  felt  it  was  a  waste  of  time. 

Project  3 

Most  felt  that  they  would  benefit  and  their  jobs  would  be  easier  to  perform. 

b.  How  did  management  react  to  the  SEI  assessment? 


RESPONDENT 

RESPONSE 

SEPG  1 

Top-level  management  has  always  been  supportive.  We’ve  had  some  fust- 
level  resistance. 

sepg: 

Management  continues  to  be  supportive  of  SEI-related  process  improvement 
efforts. 

Project  1 

Well. 

Project  2 

Very  good,  they  were  pleased  with  the  resulted. 

Project  3 

Quite  favorable.  They  are  100%  in  favor  of  improving  our  posture  in 
maintaining  software. 
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c.  How  thoroughly  has  management  embraced  the  SEI  assessment  results/suggestions 
(active  or  just  filling  a  square)? 


RESPONDENT 

RESPONSE 

SEPG1 

Our  monthly  management  steering  teams  usually  last  over  3  hours. 

SEPG  2 

Management  has  been  actively  involved  with  “pre  and  post"  assessment 
activities. 

Project  1 

Well. 

Project  2 

About  as  well  as  could  be  expected.  Change  is  not  something  that  bqtpens 
very  fast.  Management  is  fully  behind  SEI  though. 

They  are  actively  participating  in  the  process  that  need  improvement 

d.  What,  if  any,  negative  effects  have  been  realized  as  a  result  of  the  SEI  assessment? 


RESPONDENT 

RESPONSE 

SEPG  1 

none. 

SEPG2 

none. 

Project  1 

A  little  bit  of  feeling  that  SEI  doesn't  help  us  very  much. 

Project  2 

Note  that  I  can  see  yet. 

E3S39H1 

None  that  I  am  aware  of. 

e.  Has  the  non-attributive  environment  been  maintained?  (or  has  head-hunting  taken 
place) 


RESPONDENT 

RESPONSE 

SEPG  1 

We  don't  target  people  but  we  do  target  specific  projects.  I  don't  think  people 
have  a  problem  with  that  if  their  problems  get  solved 

SEPG  2 

Management  key  project  people  and  workers  have  not  pointed  the  finger  at 
other  personnel  associated  with  some  of  the  assessment  Findings. 

No  head-hunting  has  taken  place. 

uzszstamM 

Has  been  maintained. 

E3S5SNMI 

The  non-attributive  environment  has  been  maintained. 

f.  Has  the  status  of  these  factors  changed  since  your  process  improvement  efforts  were 
initiated? 


RESPONDENT 

RESPONSE  1 

SEPG  1 

There  has  been  some  resistance  but  the  longer  we  go  the  less  we  see. 

SEPG  2 

No. 

N/A 

Eg S  &WMHK 

Not  sure  which  factors  you're  talking  about 

Too  early  to  tell. 

Append  xC:  ALC  Number  2 

This  appendix  contains  a  complete  listing  of  the  questionnaire  responses  for  ALC 
Number  2. 


Section  1 .  Motivation  for  Process  Evaluation! Improvement  ( ALC  Level) 

a.  What  factors  led  to  the  initiation  of  a  process  improvement  program? 


RESPONDENT 

RESPONSE 

SEPG  1 

Mandate 

SEPG2 

A  couple  of  years  before  the  AF  ECS  PMP,  our  branch  chief  read  seme  papers 
on  process  improvement  and  what  was  going  on  at  the  SEI.  He  thought  it 
sounded  like  a  good  idea,  so  he  had  us  get  involved.  We  did  a  self  assessment 
at  the  time  (Follow-Up  Conversation  -  there  was  never  any  action  taken  as  a 
result  of  this  7  Nov  90  internal  assessment) 

RMlWWIli 

N/A 

E25SOHHH 

A  need  to  cut  costs  to  become  competitive 

b.  What  factors  led  specifically  to  SEI  involvement? 


RESPONDENT 

RESPONSE 

SEPG  1 

I  don't  know 

SEPG  2 

The  AFECS  PMP  initiative 

Project  1 

N/A 

Project  2 

? 

c.  Wbat  is  your  overall  opinion  of  the  SEI  assessment  process/mcthodology? 
(1)  Advantages 


RESPONDENT 

RESPONSE 

SEPG  1 

Conceptual  framework  and  educated,  helpful  people 

SEPG  2 

Organize  the  evaluation  process  and  define  the  evaluation  criteria 

ElSSSSHI^H 

N/A 

ESSEBMH 

Good,  a  thorough  job  done  in  a  brief  period 

(2)  Disadvantages 


RESPONDENT 

RESPONSE 

SEPG  1 

Distance 

SEPG  2 

Systems  engineering  is  missing.  Human  resources  also.  Takes  a  long  time  to 
show  change  to  management 

ES5D— 

N/A 

G553SHNI 

Don't  think  our  customer  realized  how  important  their  participation  was 
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d.  Is  maturity  level  viewed  as  a  legitimate  representation  of  your  organization's  ability 
to  produce/support  software? 


RESPONDENT 

RESPONSE 

SEPG1 

Yes 

SEPG2 

No,  sometimes  the  customer  does  not  want  to  incur  the  cost  Could  go  for 
yean  without  a  change  in  level 

Project  1 

N/A 

Yes 

e.  For  which  reasons  is  it  valuable  to  strive  for  higher-level  ratings?  (Circle  all  that  apply) 

(1)  Perception  (of  improvement) 

(2)  Improved  Product  (Quality,  Cost,  etc.) 

(3)  Headquarters  Mandate  (Required  to  stay  in  business) 

(4)  None/Other 

Explain  your  answer 


RESPONDENT 

RESPONSE 

SEPG  1 

(2)  Improved  Product  -  we  know  we  need  to  improve  our  capability,  (3) 
Headquarters  mandate  makes  it  essentia] 

SEPG2 

(2)  An  example,  we  delivered  a  piece  of  software  in  which  the  tape  could  not 
generate  the  executable  (configuration  management) 

Project  1 

(2)  and  (3).  We  need  higher-level  ratings,  not  to  say  ware  are  a  level  11  or  HI, 
but  because  we  improve  ourselves  at  each  level 

Project  2 

(2)  Quality  and  cost  of  our  product  is  more  important  than  mandates  and 
perceptions 

Section  2.  General  Process  Improvement! Assessment  Issues  (ALC  and 
Project  Level) 


a.  Was  the  evaluation  accomplished  entirely  by  the  SEI,  through  an  SEI-assistcd  team,  or 
a  self-assessment? 


RESPONDENT 

RESPONSE 

SEPG  1 

SEI-assisted  team 

SEPG  2 

SEI-assisted  team 

Project  1 

The  evaluation  was  conducted  by  an  SEI-assisted  team 

nsssamm 

SEI-assisted  team 
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b.  What  process  improvcments/actions  have  been  taken  solely  as  a  result  of  the 
assessment  process? 


RESPONDENT 

RESPONSE  1 

SEPG1 

1.  Established  management  steering  council  for  process  improvement 

2.  Established  executive  center  group  for  customer  forum 

3.  Developed  written  software  engineering  policy 

SEPG  2 

1.  Action  teams  started  on  assessment  results 

2.  Strategic  and  tactical  plans  in  development 

3.  Project  Reviews 

4.  Reorganization 

5.  Software  engineering  policy  statement 

Project  1 

We  have  looked  at  our  process.  We  have  started  documenting  and  have  taken 
steps  to  improve  program  management  and  testing  methods 

Project  2 

1.  Implemented  TPS  development  guide 

2.  Implemented  new  procedures  to  track  costs 

c.  What  other  process  improvements/changes  have  been  undertaken  within  the  same  time 
frame? 


RESPONDENT 

RESPONSE 

SEPG  1 

1.  Project  reviews 

2.  Reorganization  to  establish  formal  SQA,  CM,  and  independent  test  and 
evaluation 

SEPG2 

Several  steering  councils  hare  been  formed 

Project  1 

Improved  Software  change  control  process  (i.e.  subm itting/approval  of  Form 

75) 

Reporting  of  projects  more  frequent  to  our  customers 

d.  What  benefits  have  been  realized  through  the  SEI  Assessment  Process? 


RESPONDENT 

RESPONSE 

SEPG  1 

Increase  awareness  of  software  engineering  requirements 

SEPG  2 

Management  involvement,  especially  in  problem  resolution  and  status  reviews 

Protect  1 

Better  tracking  of  software  changes  and  improved  testing  methods 

CSS30HHI 

TPS  costs  have  been  baselined  and  high  cost  phases  have  been  identified 
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c.  What  process  problems  have  been  identified? 


RESPONDENT 

RESPONSE 

SEPG  1 

Inadequate  or  non-existent  requirements  specification,  planning  and 
management  oversight.  Virtually  no  CM  or  SQA. 

SEPG2 

Role  of  middle  management  in  improvement  effort 

Takes  a  long  time  to  get  going 

Better  defined  improvement  process  effort  definition  for  planning  purposes 

Project  1 

Need  more  team  approach  to  software  management 

Need  to  better  track  costs 

Management  needs  to  take  a  bigger  role 

Project  2 

Customer  not  involved  with  process 

Certain  phases  of  TPS  development  were  high  cost 

Section  3.  Impact  on  Project  Success  (ALC  and  Project  Level ) 


a.  At  what  level  was  the  organization  rated? 


RESPONDENT 

RESPONSE 

SEPG  1 

Level  I 

SEPG  2 

Level  I 

BSSSDBii 

Level  I 

Level  I 

b.  What  actions  taken  as  a  result  of  the  SEI  assessment(s)  have  been  part  of  this  project? 


RESPONDENT 

RESPONSE 

SEPG  1 

N/A  -  however,  a  number  of  projects  we  have  observed  have  increased 
emphasis  on  SOWs  and  CM. 

SEPG  2 

N/A 

Project  1 

Project  1  tends  to  be  the  guinea  pig  for  any  changes/invcstigations  that  the 
SEPG  need. 

Project  2 

Implemented  TPS  development  guide  to  have  repeatable  results  across  the 
project 

*• 
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c.  How  successful  was  (is)  this  project?  (Please  address  the  following  areas) 

£ 

i  (1)  Customer  satisfaction 

i  (2)  Cost 

[  (3)  Schedule 

i  (4)  Other  areas? 


RESPONDENT 

RESPONSE 

SEPG1 

Too  early  to  tell 

SEPG2 

N/A 

Project  1 

1 .  customers  are  more  satisfied  because  we  are  able  to  produce  better  quality 
softwae. 

2.  The  costs  are  being  tracked  now,  so  we  can  better  estimate  future 
software  releases.  We  can  not  compare  software  releases  and  their  costs 
(before  we  had  no  way  of  doing  so) 

3.  Our  schedules  arc  being  cracked  and  we  can  now  set  realistic  time  lines 
for  future  software  releases 

Project  2 

Customer  satisfaction  seems  to  be  high  especially  since  we  can  more 
effectively  predict  costs  and  schedule 

<L  How  have  the  post-SEI  assessment  actions  affected  the  success  of  this  project  in  the 
following  areas?  (Please  address  the  following  areas) 

(1)  Customer  satisfaction 

(2)  Cost 

(3)  Schedule 

(4)  Other  areas? 


iimmw 

RESPONSE 

SEPG  1 

Marginally  to  date  -  although  management  interest  in  project  reviews  have 
helped,  especially  when  management  offers  to  help  resolve  problems 

SEPG2 

N/A 

Project  1 

See  3c  for  answer 

IGStSaMI 

See  3c  for  answer 

e.  Do  you  feel  the  proposed  improvements  will  directly  result  in  benefits  to  this  project? 
Why? 


RESPONDENT 

RESPONSE 

SEPG  1 

Can't  really  comment  on  specific  project 

SEPG  2 

N/A 

Project  1 

Yes.  If  the  schedules  and  costs  can  be  predictable,  and  if  our  process  can  be 
"standardized"  (formalized),  the  customer  will  be  satisfied  ■  which  should 
always  be  our  goal. 

Yes,  higher  quality  and  lower  costs  of  TPSs  will  be  the  big  benefit 
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Section  4.  Hum*  n  factors  (ALC  and  Project  Level) 

a.  What  was  the  worker  level  feeling  concerning  the  SEI  assessment? 


RESPONDENT 

RESPONSE 

SEPG  1 

Skepticism  followed  by  surprise  at  final  report.  Currently  taking  a  wait  and 
see  approach.  Progress  has  been  slow 

SEPG2 

Lots  of  good  stuff,  but  needs  to  be  realistic/usable  (more  paperwork  must  show 
added  value,  SQA  should  not  just  be  a  check  list  group,  more  costly  to 

improve  can  we  remain  competitive?)  concern  on  management  oversight 

Project  1 

Just  another  formality,  nothing  would  come  out  of  it 

Project  2 

They  were  viewed  as  another  inspection  by  folks  who  were  self-proclaimed 
experts 

b 

How  did  management  react  to  the  SEI  assessment? 

RESPONDENT 

RESPONSE 

SEPG  1 

For  the  most  part,  constructively 

SEPG  2 

Check  to  box,  so  we  can  get  to  level  n,  in  the  beginning.  They  are  slowly 
changing 

Project  1 

The  were  all  behind  it  Management  always  gets  behind  these  kinds  of  things 
-  it's  their  follow  through  that  makes  people  skeptical.  In  other  words  -  they 
usually  don't  do  any  follow  through. 

Project  2 

Something  that  had  to  be  done  and  the  outcome  (Level  I)  was  not  surprising 

c.  How  thoroughly  has  management  embraced  the  SEI  assessment  results/suggestions 

(active  or  just  filling  a  square)? 

RESPONDENT 

RESPONSE 

SEPG  1 

Some  see  value.  To  others,  it  is  just  another  fad. 

SEPG  2 

Slowly  becoming  active.  Our  forming  of  the  management  steering  council 
has  really  helped  to  get  their  involvement. 

Project  1 

It  was  active  for  a  while,  now  some  managers  are  just  going  through  the 
motions 

Project  2 

Active,  but  their  plans  are  slow  to  implement 

d.  What,  if  any,  negative  effects  have  been  realized  as  a  result  of  the  SEI  assessment? 

RESPONDENT 

RESPONSE 

SEPG  1 

Fear  of  unproductive  management  involvement 

SEPG  2 

Hidden  agendas  are  coming  out  in  an  attempt  to  use  the  improvement  effort  as 
a  piggyback  mechanism 

Project  1 

Workers  are  upset  that  we  wanted  the  assessment  to  mean  something,  but 
overall  there  have  not  been  many  changes. 

Project  2 

We  are  more  concerned  about  satisfying  the  SEPG,  than  about  our  customer 
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e.  Has  the  non-attributive  environment  been  maintained?  (or  has  head-hunting  taken 
place) 


RESPONDENT 

RESPONSE 

SEPG  1 

No  head-hunting 

SEPG2 

Practitioners  and  managers  are  still  very  concerned  about  non-attribution. 
(Practitioners  are  also  concerned  about  advancement  if  they  get  involved  in  a 
long-term  improvement  effort  that  uses  50%  of  their  time) 

Project  1 

For  the  most  part 

Project  2 

Yes 

f.  Has  the  status  of  these  factors  changed  since  your  process  improvement  efforts  were 
initiated? 


RESPONDENT 

RESPONSE 

SEPG  1 

Tear  has  decrease,  but  skepticism  about  value  of  process  improvements 
remains. 

SEPG  2 

N/A 

Project  1 

No,  no  really.  Management  will  support  most  anything,  especially  if  they 
don't  have  to  do  anything  about  it. 

Project  2 

No 
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Appendix  D:  ALC  Number  3 

This  appendix  contains  a  complete  listing  of  the  questionnaire  responses  for  ALC 
Number  3. 

Section  1.  Motivation  for  Process  Evaluation! Improvement  ( ALC  Level) 

a.  What  factors  led  to  the  initiation  of  a  process  improvement  program? 


RESPONDENT 

RESPONSE 

SEPG  1 

It  was  mandated  by  the  Air  Force.  If  it  hadn't  been  mandated,  it  wouldn't 
have  been  done. 

SEPG2 

SEI  self-assessment  in  June  1992  indicated  that  no  SQA  function  was  present 

SEPG  3 

Directorate  commitment 

SEPG  4 

N/A 

SEPG  5 

Don't  know 

E235I9Q0HI 

N/A 

N/A 

G323EWM1 

Mandated  by  management 

cawr— | 

The  need  to  standardize,  which  identified  areas  of  duplication 

Trying  to  cut  costs  and  complete  schedule  r  itad  of  time 

b.  What  factoi '  led  specifically  to  SEI  involvement? 


RESPONDENT 

RESPONSE 

SEPG  1 

It  was  mandated  by  the  Air  Force.  Nobody  formally  evaluated  the  advantages 
and  disadvantages  of  SEI  involvement. 

SEPG  2 

The  SEPG  had  the  help  of  the  SEI  in  their  assessment  (June  1992) 

SEPG  3 

1.  HQ  AF  mandate 

2.  Directorate/Divisicn  decision  to  measure  current  state 

SEPG  4 

N/A 

SEPG  5 

? 

N/A 

N/A 

Mandated  by  management 

ESSXsBBBMI 

The  need  to  be  more  competitive,  hold  costs  down. 

Project  D 

Our  division  know  >  we  must  be  competitive.  SEI  is  standard  private  industry 
uses. 
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c.  What  is  your  overall  opinion  of  the  SEI  assessment  process/methodology? 
(1)  Advantages 


RESPONDENT 

RESPONSE 

SEPG  1 

If  used,  the  CMM  provides  the  correct  methodology  for  managing  a  project, 
not  just  a  software  project.  Involving  SEI  in  an  assessment  means  that  the 
assessed  organization  benefits  from  SETs  experience  and  obtains  a  legitimate 
assessment 

SEPG2 

The  opinions  come  from  the  working  level 

SEPG  3 

Consistent 

SEPG  4 

N/A 

SEPG  5 

It  is  a  set  method 

N/A 

E355E5^HBI 

N/A 

Project  B 

It  allows  us  to  better  define  requirements  and  sets  a  well-defined  path  in 
which  to  follow  in  our  process 

N/A 

ESSI3HH 

I  feel  it  will  show  us  our  weaknesses  so  we  can  improve 

(2)  Disadvantages 


RESPONDENT 

RESPONSE 

SEPG  1 

The  CMM  isn't  a  cookbook  and  it  isn't  all  that  easy  to  understand.  The 
project  managers  must  spend  considerable  time  studying  it  before  they  can 
implement  it  in  their  projects. 

SEPG  2 

Management  can  justify  the  position  of  the  organization  and  paint  a  rosy 
picture. 

SEPG  3 

Some  criteria  very  vague  to  subjective 

SEPG  4 

N/A 

SEPG  5 

Does  not  address  systems  (hardware) 

ESfiSSESEHU 

N/A 

ESSBHSBBB 

N/A 

Project  B 

We  have  a  tendency  to  improve  things  to  the  point  that  they  don't  work  after 
the  improvement  (KISS  -  keep  it  simple  sir) 

N/A 

N/A 

4> 
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d.  Is  maturity  level  viewed  as  a  legitimate  representation  of  your  organization's  ability 
to  produce/support  software? 


RESPONDENT 

RESPONSE 

SEPG  1 

Both  management  and  the  workers  put  heavy  stock  in  maturity  levels,  but 
they  tend  to  over  estimate  their  own  maturity  levels.  Management's  estimates 
are  higher  than  the  worker's. 

SEPG2 

It  definitely  is  a  metric  to  be  considered  -  but  not  solely 

SEPG  3 

More  as  a  measure  of  consistency,  repeatability,  and  predictability 

SEPG  4 

N/A 

SEPG  5 

Yes 

E2R513BB1 

N/A 

E323H3MB 

N/A 

Project  B 

I  feel  that  for  the  most  part  it  does.  Except  that  we  may  not  be  as  formalized 
and  documented  as  we  should  be. 

Project  C 

What  is  maturity?  The  idea  does  not  represent  a  company's  ability  to  be 
profitable  and  perform  work  for  the  customer 

Yes 

c.  For  which  reasons  is  it  valuable  to  strive  for  higher-level  ratings?  (Circle  all  that  apply) 

(1)  Perception  (of  improvement) 

(2)  Improved  Product  (Quality,  Cost,  etc.) 

(3)  Headquarters  Mandate  (Required  to  stay  in  business) 

(4)  None/Other 

Explain  your  answer.  • 


RESPONDENT 

RESPONSE 

SEPG1 

For  me  personally,  it  is  only  to  improve  the  product  Both  as  an  aircraft 
software  design  engineer  and  as  the  MEG  chairman,  I  worry  about  our 
software's  quality;  but  for  many  division  employees,  it  is  simply  mandated  by 
the  Air  Force.  If  it  hadn't  been  mandated,  it  wouldn't  have  been  done.  As  a 
member  of  the  SQA  team,  I  feel  a  heavy  obligation  to  help  institutionalize  the 
CMM. 

SEPG2 

The  higher  level  ratings  should  show  improved  efficiency  and  quality  of  the 
end  product 

SEPG3 

1 .  To  improve  morale 

2.  Competitiveness  and  less  maintenance 

3.  Self-explanatory 

SEPG4 

N/A 

SEPG5 

N/A 

EsrimnF 

N/A 

ESSEBHH 

N/A 

Project  B 

Improved  product  and  headquarters  mandate.  We  are  all  professionals,  which 
means  that  quality  and  cost  are  the  only  options  to  stay  in  business 

Project  C 

Headquarters  mandate.  A  committed  level  in  organization  should  be 
adequate 

Project  D 

Improved  Product  By  improving  our  processes,  we  cut  costs  and  schedules 
plus  put  out  a  better  product 
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Section  2.  General  Process  Improvement! Assessment  Issues  (ALC  and 
Project  Level) 


a.  Was  the  evaluation  accomplished  entirely  by  the  SEI,  through  an  SEI-assisted  team,  or 
a  self-assessment? 


RESPONDENT 

RESPONSE 

SEPG  1 

An  SEI  assisted  team.  This  approach  has  two  advantages.  First,  SEI 
involvement  brings  SETs  vast  experience  to  the  assessment  Second,  having 
employees  from  the  assessed  organization  on  the  assessment  team  gives  the 
organization  insight  and  a  buy-in  into  the  assessment  process 

SEPG  2 

It  was  a  self-assessment,  but  it  had  representation  from  the  SEI  and  other 

ALCs 

SEPG  3 

SEI-assisted  team 

SEPG  4 

SEI-assisted  team  with  members  from  the  other  four  ALCs 

SEPG  5 

SEI-assisted  team 

Project  A-l 

Local  SEPG  assessment  I  was  not  sure  about  this,  but  asked  someone  else 
that  knew. 

E33EHI 

Self-assessment  by  a  local  SEPG  team 

I  think  though  an  SEI-assisted  team 

Assessment  team 

Through  a  SEI-assisted  team  and  self  assessment 

b.  What  process  improvements/actions  have  been  taken  solely  as  a  result  of  the 
assessment  process? 


RESPONDENT 

RESPONSE 

SEPG  1 

Organized  eighteen  Implementation  Teams  to  determine  the  actions  each 
project  must  perform  to  satisfy  all  of  the  Level  II  and  Level  IU  KPAs 

SEPG  2 

SQA  team  formed  - 18  implementation  teams  formed  from  working  level 
employees 

SEPG  3 

SEPG  and  independent  SQA  groups  formed 

SEPG  4 

We  have  initiated  approximately  18  improvement  efforts 

SEPG  5 

Implementation  teams  have  been  organized  for  each  of  the  KPAs  in  Levels  II 
and  III  of  the  CMM 

Project  A-l 

More  formal  peer  reviews  have  been  encouraged 

Project  A-2 

Peer  reviews  have  been  pushed  by  management 

Project  B 

We  now  track  our  peer  reviews  where  before,  we  would  hold  them  but  no 
record  was  kept. 

E3S33 

An  attitude  shift  toward  the  SEI  standards 

Better  peer  reviews,  weekly  status  meetings 
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c.  What  other  process  improvements/changes  have  been  undertaken  within  the  same  time 
frame? 


RESPONDENT 

RESPONSE 

SEPG  1 

None. 

SEPG2 

Peer  reviews  have  been  standardized,  and  process  definition  has  begun. 

SEPG  3 

Process  baselining,  metrics  enhancement 

SEPG  4 

None. 

SEPG  5 

Human  resources,  software  metrics,  management  reviews,  test  equipment, 
project  resources,  organizational  policies  and  procedures 

Project  A- 1 

Configuration  management  process  has  been  improved  and  is  continuing  to 
be  improved 

Project  A-2 

Configuration  management  of  software  process  has  been  evaluated  and 
improved 

Project  B 

We  work  closer  with  our  customers.  We  attend  a  team  meeting  with  them 
twice  a  month. 

The  formation  of  implementation  teams 

Using  a  project  management  program  to  track  each  task 

d.  What  benefits  have  been  realized  through  the  SEI  Assessment  Process? 


RESPONDENT 

RESPONSE 

SEPG  1 

The  SQA  team  and  the  implementation  teams  have  been  organized  and  are 
functioning.  Other  than  that,  it's  too  soon  to  determine  the  impact  these 
teams  will  have. 

SEPG  2 

The  net  resuiis  haven't  really  surfaced  yet  -  were  still  in  the  pain  stage 

SEPG  3 

Process  awareness,  metrics  awareness 

SEPG  4 

Increased  awareness  of  the  need  for  process  improvement  within  the 
organization.  Helped  to  focus  process  improvement  effort. 

SEPG  5 

Implementation  teams  organized. 

Project  A-l 

Nothing  specific,  other  than  a  vision  for  change  and  more  (formal)  peer 
reviews 

Project  A-2 

Clear  understanding  of  our  own  processes  and  looking  for  ways  to  improve 
them.  Changing  and  evaluating  the  process  improves  quality 

Eu355EBHHi 

We  try  to  solve  stumbling  blocks  before  they  develop  into  real  problems 

Eu3SS3E3NNi 

Awareness  of  SEI 

Better  understanding  by  everyone  -  what  our  processes  are. 
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e.  What  process  problems  have  been  identified? 


RESPONDENT 

RESPONSE 

SEPG  1 

None  yet 

SEPG2 

Peer  reviews  are  not  consistent  -  SQA  is  non-existent 

SEPG  3 

Configuration  management  of  processes 

SEPG  4 

Refer  to  the  assessment  report  findings  and  recommendations 

SEPG  5 

Listed  in  questions  2b  and  2c 

ESSEuflHH 

The  need  to  identify  a  specific  process 

Project  A-2 

First  major  problem,  people  don't  know  what  the  process  is  and  that  it  can  be 
changed 

ESSSIBHH 

I  feel  that  our  processes  are  solid 

EsmaMi 

N/A 

j  Project  D 

We  haven’t  allowed  enough  time  for  some  areas  of  our  processes,  (i.e. 
fabrication,  lead  time  for  parts) 

Section  3.  Impact  on  Project  Success  (ALC  and  Project  Level) 
a.  At  what  level  was  the  organization  rated? 


RESPONDENT 

RESPONSE 

SEPG  1 

Emerging  Level  II.  As  far  as  I’m  concerned,  that's  a  polite  way  of  saying 

Level  I. 

SEPG  2 

Emerging  Level  11.  (by  the  old  standards) 

SEPG  3 

Emerging  Level  HI 

SEPG  4 

Emerging  Level  11,  e.g.  some  of  the  Level  II  and  Level  in  KPAs  satisfied,  but 
not  all. 

SEPG  5 

Division 

E3£3E3QHfll 

Level  I 

C3SB20BBB 

One 

Level  I-II 

1.8 

Level  II 

i 
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b.  What  actions  taken  as  a  result  of  the  SEI  assessment(s)  have  been  part  of  this  project? 


RESPONDENT 

RESPONSE 

SEPG  1 

The  SQA  team  and  the  implementation  teams  have  been  organized  and  are 
functioning. 

SEPG2 

SQA  team  formed  -  with  the  goal  of  ensuring  SQA  functions  are  performed  as 
part  of  the  development  cycle. 

SEPG  3 

Better  SQA,  process  configuration  management 

SEPG  4 

We  are  implementing  the  activities/functions  necessary  to  satisfy  the  SQA 

KPA  goals. 

SEPG  5 

Organization  of  implementation  teams 

Project  A- 1 

None  that  I  know  of. 

Project  A-2 

None 

Project  B 

? 

Project  C 

Peer  reviews  initiated 

Project  D 

As  a  result  of  the  assessment,  we  started  using  a  project  management  program 

c.  How  successful  was  (is)  this  project?  (Please  address  the  following  areas) 

(1)  Customer  satisfaction 

(2)  Cost 

(3)  Scliedule 

(4)  Other  areas? 


RESPONDENT 

RESPONSE  \ 

SEPG  1 

The  SQA  team  and  implementation  teams  take  people  away  from  the  projects. 

It  is  too  soon  to  determining  what  impact  this  will  have  on  quality. 

SEPG  2 

The  team  was  just  formed  two  months  ago  -  the  attitude  ranges  from 
encouraging  to  neutral  throughout  the  division  at  this  time. 

SEPG  3 

1.  Good 

2.  Fair 

3.  Very  Good 

SEPG  4 

Too  early  to  tell  right  now.  We  just  started  this  project/effort  in  May  93. 

SEPG  5 

Not  far  enough  to  know. 

Project  A-l 

Customer  satisfaction  is  good,  cost  is  within  budget,  and  schedule  is  on-target 

Project  A-2 

The  customer  is  satisfied  with  product.  The  project  is  within  budget  and  on 
schedule. 

Project  B 

We  now  work  much  closer  to  our  customer.  Cost  and  schedule  have  had  no 
change  this  close  to  the  end  of  the  assessment.  We  have  better  equipment  to 
do  our  jobs. 

Project  C 

Customer  satisfaction  is  good  and  we  are  able  to  manage  costs.  Our  schedule 
lias  been  blown  out  of  the  water  by  contractor  involvement 

Project  D 

Customer  is  pleased,  cost  is  average  for  the  complexity  of  these  programs 
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d.  How  have  the  post-SEI  assessment  actions  affected  the  success  of  this  project  in  the 
following  areas?  (Please  address  the  following  areas) 

(1)  Customer  satisfaction 

(2)  Cost 

(3)  Schedule 

(4)  Other  areas? 


RESPONDENT 

RESPONSE 

SEPG  1 

It  is  too  soon  to  tell.  As  the  SQA  team,  we  need  to  develop  metrics  to  show 
how  the  software  quality  improves  as  the  KPAs  are  implemented.  I'm  not 
sure  how  we  do  this  yet. 

SEPG2 

Same  as  3c 

SEPG  3 

1.  No  change 

2.  Higher 

3.  No  change 

SEPG  4 

See  question  3c 

SEPG  5 

It  got  us  started 

Project  A-l 

It  has  not  directly  affected  customer  satisfaction,  cost,  or  schedule. 

Project  A-2 

The  SLI  assessment  has  not  really  affected  any  of  these  areas 

ESSS5EBMH 

See  3c 

ESnSHH 

N/A 

Project  D 

It  showed  us  what  we  were  doing  could  be  improved  by  showing  customers 
the  status  of  their  projects.  It  also  showed  how  we  might  cut  costs  and 
improve  schedules. 

95 


e.  Do  you  feel  the  proposed  improvements  will  directly  result  in  benefits  to  this  project? 
Why? 


RESPONDENT 

RESPONSE 

SEPG  1 

Definitely,  I  strongly  believe  the  CMM  provides  the  structure  to  manage  any 
project  The  closer  the  division  follows  the  CMM,  the  more  successful  it  will 
be. 

SEPG2 

This  project  hopes  to  result  in  benefits  to  the  developmental  units  in  the 
division  as  SQA  is  implemented 

SEPG  3 

Yes.  Improved  quality  and  better  design 

SEPG  4 

Yes 

SEPG  5 

This  project  is  an  improvement  effort 

Project  A-l 

This  project  will  be  completed  by  the  time  any  proposed  improvements  would 
affect  it  They  may  benefit  the  use  and  maintenance  of  the  station  developed 
by  this  project 

Project  A- 2 

I  think  SEI  assessment  has  had  us  look  at  our  processes,  but  I  can't  see 
anything  that  will  improve  or  benefit  our  project  that  we  were  not  already 
addressing 

Project  B 

Yes,  only  if  management  will  mandate  all  to  get  involved  in  the  improvement. 
The  few  can  not  move  us  to  the  next  higher  level  alone. 

ES3SSSMH 

To  establish  a  method  and  procedure  for  handling  the  unexpected. 

Project  D 

Yes,  because  we  have  better  tracking  in  place,  and  everyone  on  the  project 
understands  the  part  they  play. 

Section  4.  Human  Factors  ( ALC  and  Project  Level) 


a.  What  was  the  worker  level  feeling  concerning  the  SEI  assessment? 


RESPONDENT 

RESPONSE 

SEPG  1 

It  was  positive.  The  workers  believe  that  the  assessment  was  realistic  and 
they  support  the  implementation  teams.  They  think  the  assessment  is  one  way 
to  get  management  to  listen  to  their  problems.  The  SQA  team  has  an 
excellent  relationship  with  the  engineers  because  we  worked  with  them  for 
many  years. 

SEPG  2 

Some  think  it  will  go  away  like  other  government  programs,  others  are 
grasping  onto  the  principles 

SEPG  3 

Good 

SEPG  4 

Not  sure  what  the  benefit/result  of  the  assessment  would  be 

SEPG  i 

Most  were  unaware  or  uninformed 

Not  well  informed  about  the  SEI  assessment  or  how  it  related  to  this  project 

KZSftSESSHH 

We  got  a  Level  I,  now  what? 

Project  B 

there  were  not  many  involved.  Input  was  not  given.  I  feel  that  the  feeling 
was  just  "its  another  quality  program  being  forced  upon  us" 

ES53sMH 

They  have  a  cautious  idea  of  the  SEI  (A  wait  and  see  attitude) 

Project  D 

They  now  feel  like  a  better  part  of  the  team  and  lealize  everyone  must  do  their 
share 

b.  How  did  management  react  to  the  SEI  assessment? 


RESPONDENT 

RESPONSE 

SEPG  1 

They  took  it  very  seriously  and  understood  they  must  create  the  corporate 
culture  required  to  satisfy  the  KPAs 

SEPG2 

All  managers  think  it  was  their  section  who  is  Level  III  -  to  bring  the  rest  of 
the  division  up  to  Emerging  Level  II  (a  very  scary  assumption) 

SEPG  3 

Responded  with  improvement  plans 

SEPG  4 

Used  it  as  a  tool  to  provide  insight  on  where  to  focus  our  process  improvement 
efforts. 

SEPG  5 

Business  as  usual 

Project  A-l 

Management  has  begun  to  discuss  the  need  to  improve  and  be  a  Level  III  by 
next  spring. 

Project  A-2 

We  got  to  get  a  better  level  next  time  -  management  hasn’t  said  much  since 
the  fust  few  weeks  after  the  assessment. 

Project  B 

I  think  some  management  had  a  good  reaction,  but  there  are  still  many  that 
support  it  just  to  satisfy  the  requirement.  I  don’t  see  any  push  of  excitement. 

ESBSOBB 

Some  felt  cheated,  otheis  were  relieved. 

Project  D 

Disappointed  we  weren't  at  a  higher  level.  But  determined  to  do  what  they 
have  to  tot  get  a  higher  level. 

c.  How  thoroughly  has  management  embraced  the  SEI  assessment  results/suggestions 
(active  or  just  tilling  a  square)? 


RESPONDENT 

RESPONSE 

SEPG  1 

Management  appears  to  be  totally  committed  to  implementing  the  SEI 
assessment  suggestion.  They  are  more  serious  than  just  filling  a  square. 

SEPG  2 

Because  the  division  has  committed  to  become  Level  III  by  August  1994,  the 
management  is  taking  a  more  serious  approach  than  in  the  past. 

SEPG  3 

Active,  but  slow 

SEPG  4 

Seem  to  have  fully  embraced  them 

SEPG  5 

Filling  a  square 

Project  A-l 

I  feel  management  has  or  is  beginning  to  embrace  the  results/suggestions,  but 
lacks  understanding  on  how  to  get  the  process  functioning. 

Project  A-2 

Just  filling  the  square.  They  push  the  ideas  and  suggestions,  but  we  receive 
no  training  or  help  in  implementing  them,  except  for  a  1-hour  class  on  peer 
reviews 

See  3b 

Project  C 

Yes  and  no.  Most  are  attempting  to  initiate  SEI  standards  and  others  are 
filling  squares 

Project  D 

Management  is  very  actively  involved  in  us  reaching  our  goals 
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d.  What,  if  any,  negative  effects  have  been  realized  as  a  result  of  the  SEI  assessment? 


RESPONDENT 

RESPONSE 

SEPG  1 

We've  just  changed  division  chiefs.  The  old  division  chief,  who  is  not  the 
department  head,  held  the  assessment's  final  report  because  he  didn't  want  the 
consequences  included  in  the  findings.  The  new  division  chief  doesn't  appear 
to  have  these  problems. 

SEPG2 

None  - 1  believe  everyone  thinks  there  is  always  room  for  improvement  -  just 
how  far  management  will  support  it  -  only  time  will  tell. 

SEPG  3 

Unknown 

SEPG  4 

Some  people  felling  overwhelmed  with  the  process  improvement  efforts  that 
have  resulted  from  the  assessment 

SEPG  5 

The  result  was  an  "emerging  Level  II."  We  were  really  Level  I  and  this  gave 
overconfidence 

None  that  I  know  of 

Project  A-2 

The  concern  appears  to  be  in  getting  a  higher  level  in  the  next  assessment, 
rather  than  seeing  how  the  model  applies  to  a  project  and  see  if  it  will  ad 
value  to  it. 

I  think  that  not  enough  training  and  motivation  have  been  pushed 

Project  C 

None 

Project  D 

N/A 

e.  Has  the  non-attributive  environment  been  maintained?  (or  has  head-hunting  taken 
place) 


RESPONDENT 

RESPONSE 

SEPG  1 

Yes 

SEPG  2 

No  Itead-hundng  has  occurred,  but  management  is  denying  that  certain 
realities  did  in-fact  surface. 

SEPG  3 

Yes 

SEPG  4 

Yes 

SEPG  5 

Yes 

Project  A-l 

Have  not  seen  any  head-hunting,  assume  a  non-attributive  environment  has 
been  maintained 

Yes 

E335EMBB1 

Yes 

Yes,  the  head-hunter's  attitude  has  not  surface  yet 

Project  D 

By  better  tracking,  you  don't  need  to  head-hunt  a  person  can  see  quickly  how 
their  performance  as  a  team  member  is. 
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f.  Has  the  status  of  these  factors  changed  since  your  process  improvement  efforts  were 
initiated? 


RESPONDENT 

RESPONSE 

srPG  i 

I  think  the  workerAnanagement  relationship  will  improve  as  they  cooperate  to 
impiment  the  CMM  and  SE1  suggestions 

SEPG2 

If  management  denies  that  a  problem  exists!  How  do  you  fix  something  that  is 
not  broken? 

SEPG3 

Morale  is  slightly  higher. 

SEPG4 

What  factors? 

SEPG5 

No 

Project  A*  1 

It  is  being  discussed  more,  but  I  feel  there  is  a  great  need  for  training  on  what 
exactly  is  expected  and  how  it  will  be  implemented.  There  is  a  lack  of 
understanding  on  the  process  and  there  is  no  road  map  on  moving  from  Level 

I  to  the  next. 

Project  A-2 

No 

Project  B 

No 

Project  C 

No 

Project  D 

N/A 
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